Learning from positive and unlabeled examples -Finite size sample bounds
Extracted source text · arXiv 2507.07354
\documentclass{article}
% if you need to pass options to natbib, use, e.g.:
% \PassOptionsToPackage{numbers, compress}{natbib}
% before loading neurips_2025
% ready for submission
% \usepackage{neurips_2025}
% to compile a preprint version, e.g., for submission to arXiv, add add the
% [preprint] option:
\usepackage[preprint]{neurips_2025}
% to compile a camera-ready version, add the [final] option, e.g.:
% \usepackage[final]{neurips_2025}
% to avoid loading the natbib package, add option nonatbib:
% \usepackage[nonatbib]{neurips_2025}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\input{macros}
\usepackage{thmtools}
\usepackage{thm-restate}
\bibliographystyle{apalike}
\title{Learning from positive and unlabeled examples -Finite size sample bounds}
% The \author macro works with any number of authors. There are two commands
% used to separate the names and addresses of multiple authors: \And and \AND.
%
% Using \And between authors leaves it to LaTeX to determine where to break the
% lines. Using \AND forces a line break at that point. So, if LaTeX puts 3 of 4
% authors names on the first line, and the last on the second line, try using
% \AND instead of \And before the third author name.
\author{%
Farnam Mansouri \\
University of Waterloo and Vector Institute \\
\texttt{f5mansou@uwaterloo.ca} \\
% examples of more authors
\And
Shai Ben-David \\
University of Waterloo and Vector Institute\\
\texttt{shai@uwaterloo.ca} \\
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \AND
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
% \And
% Coauthor \\
% Affiliation \\
% Address \\
% \texttt{email} \\
}
\begin{document}
\maketitle
\begin{abstract}
PU (Positive Unlabeled) learning is a variant of supervised classification learning in which the only labels revealed to the learner are of positively labeled instances.
%framework in which the learner has access only to positively labeled and unlabeled examples.
PU learning arises in many real-world applications. Most existing work relies on the simplifying assumptions that the positively labeled training data is drawn from the restriction of the data generating distribution to positively labeled instances and/or that the proportion of positively labeled points (a.k.a. the class prior) is known apriori to the learner. This paper provides a theoretical analysis of the statistical complexity of PU learning under a wider range of setups. Unlike most prior work, our study does not assume that the class prior is known to the learner.
%it includes analysis for both the SCAR setting and more general setups,
%and it introduces novel lower bounds for existing results for the SCAR setting.
We prove upper and lower bounds on the required sample sizes (of both the positively labeled and the unlabeled samples).
% This article studies sample complexity of PU learning for various different setups. In our study, unlike most prior work, we do not assume the learner knows the proportion of positive labeled points --known as class prior--. Moreover, most existing work relies on the simplifying assumption that the positive data is drawn from the same distribution as the true positive examples, a setting known as Selected Completely At Random (SCAR). While offering
% Most existing work relies on the simplifying assumption that the positive data is drawn from the same distribution as the true positive examples, a setting known as Selected Completely At Random (SCAR).
% In this paper, we offer a theoretical analysis of PU learning, which is one of a few studies that is conducted without the assumption that portion of positively labeled points is known to the learner; produces novel lower bounds for existing results in the literature under the SCAR setup; and provides a variety of new sample complexity bounds for settings beyond SCAR.
% This paper studies PU learning under both the SCAR assumption and more general settings, providing upper and lower bounds on the sample complexity.
% . This paper offers a theoretical analysis within this assumption with finite sample size guarantees
\end{abstract}
\input{Section-Introduction}
\input{Section-PU-setting}
% \input{NeuRIPS'2025/Section-pos}
\input{Section-PU-real}
\input{Section-PU-agnostic}
\section{Conclusion}
In conclusion, this work studies the sample complexity of PU learning in both realizable and agnostic settings, covering the SCAR setup as well as more general scenarios. We provide theoretical guarantees on finite sample complexity. Our results extend the existing literature by relaxing several restrictive assumptions that were made in previous publications, and by proving lower bounds on required sample sizes.
% While our results take important steps toward a theoretical understanding of PU learning, many fundamental questions regarding sample complexity and computational complexity of PU learning remain open.
% The development of new bounds and algorithms, including adaptations inspired by prior work, demonstrates that meaningful learning can be achieved even when covariate shifts are present for positive data.
\begin{ack}
We thank Sandra Zilles and Alireza Fathollah Pour for helpful discussions during the development of this work.
\end{ack}
\bibliography{ref.bib}
% % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
% \newpage
% \section*{NeurIPS Paper Checklist}
%%% BEGIN INSTRUCTIONS %%%
% The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: {\bf The papers not including the checklist will be desk rejected.} The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page
% limit.
% Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
% \begin{itemize}
% \item You should answer \answerYes{}, \answerNo{}, or \answerNA{}.
% \item \answerNA{} means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
% \item Please provide a short (1–2 sentence) justification right after your answer (even for NA).
% % \item {\bf The papers not including the checklist will be desk rejected.}
% \end{itemize}
% {\bf The checklist answers are an integral part of your paper submission.} They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
% The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "\answerYes{}" is generally preferable to "\answerNo{}", it is perfectly acceptable to answer "\answerNo{}" provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "\answerNo{}" or "\answerNA{}" is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer \answerYes{} to a question, in the justification please point to the section(s) where related material for the question can be found.
% IMPORTANT, please:
% \begin{itemize}
% \item {\bf Delete this instruction block, but keep the section heading ``NeurIPS Paper Checklist"},
% \item {\bf Keep the checklist subsection headings, questions/answers and guidelines below.}
% \item {\bf Do not modify the questions and only use the provided macros for your answers}.
% \end{itemize}
%%% END INSTRUCTIONS %%%
% \begin{enumerate}
% \item {\bf Claims}
% \item[] Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We provide formal statements and proofs of all contributions claimed in the abstract and introduction.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the abstract and introduction do not include the claims made in the paper.
% \item The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
% \item The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
% \item It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
% \end{itemize}
% \item {\bf Limitations}
% \item[] Question: Does the paper discuss the limitations of the work performed by the authors?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The limitations of our work are discussed in the introduction.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
% \item The authors are encouraged to create a separate "Limitations" section in their paper.
% \item The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
% \item The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
% \item The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
% \item The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
% \item If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
% \item While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
% \end{itemize}
% \item {\bf Theory assumptions and proofs}
% \item[] Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: Each formal statement begins with stating the premises under which it is claimed. Proofs not given in full in the main body are included in the appendix.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include theoretical results.
% \item All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
% \item All assumptions should be clearly stated or referenced in the statement of any theorems.
% \item The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
% \item Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
% \item Theorems and Lemmas that the proof relies upon should be properly referenced.
% \end{itemize}
% \item {\bf Experimental result reproducibility}
% \item[] Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This is a purely theoretical paper, without experiments.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
% \item If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
% \item Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
% \item While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
% \begin{enumerate}
% \item If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
% \item If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
% \item If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
% \item We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
% \end{enumerate}
% \end{itemize}
% \item {\bf Open access to data and code}
% \item[] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This is a purely theoretical paper, without experiments.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that paper does not include experiments requiring code.
% \item Please see the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item While we encourage the release of code and data, we understand that this might not be possible, so “No” is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
% \item The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (\url{https://nips.cc/public/guides/CodeSubmissionPolicy}) for more details.
% \item The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
% \item The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.
% \item At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
% \item Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
% \end{itemize}
% \item {\bf Experimental setting/details}
% \item[] Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
% \item[] Answer: \answerNA% Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This is a purely theoretical paper, without experiments.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
% \item The full details can be provided either with the code, in appendix, or as supplemental material.
% \end{itemize}
% \item {\bf Experiment statistical significance}
% \item[] Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This is a purely theoretical paper, without experiments.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
% \item The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
% \item The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
% \item The assumptions made should be given (e.g., Normally distributed errors).
% \item It should be clear whether the error bar is the standard deviation or the standard error of the mean.
% \item It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96\% CI, if the hypothesis of Normality of errors is not verified.
% \item For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
% \item If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
% \end{itemize}
% \item {\bf Experiments compute resources}
% \item[] Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This is a purely theoretical paper, without experiments.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not include experiments.
% \item The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
% \item The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
% \item The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
% \end{itemize}
% \item {\bf Code of ethics}
% \item[] Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics \url{https://neurips.cc/public/EthicsGuidelines}?
% \item[] Answer: \answerYes{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: This is a purely theoretical paper, without any anticipated harm or societal impact of any kind.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
% \item If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
% \item The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).
% \end{itemize}
% \item {\bf Broader impacts}
% \item[] Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: We do not anticipate any broader societal impact.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that there is no societal impact of the work performed.
% \item If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
% \item Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
% \item The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
% \item The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
% \item If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
% \end{itemize}
% \item {\bf Safeguards}
% \item[] Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: Our research does not involve data or predictive/generative models, and does not pose any risks of the mentioned kind.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper poses no such risks.
% \item Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
% \item Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
% \item We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
% \end{itemize}
% \item {\bf Licenses for existing assets}
% \item[] Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
% \item[] Answer: \answerNA% Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The paper does not use existing assets.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not use existing assets.
% \item The authors should cite the original paper that produced the code package or dataset.
% \item The authors should state which version of the asset is used and, if possible, include a URL.
% \item The name of the license (e.g., CC-BY 4.0) should be included for each asset.
% \item For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
% \item If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, \url{paperswithcode.com/datasets} has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
% \item For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
% \item If this information is not available online, the authors are encouraged to reach out to the asset's creators.
% \end{itemize}
% \item {\bf New assets}
% \item[] Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
% \item[] Answer: \answerNA% Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The paper does not release new assets.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not release new assets.
% \item Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
% \item The paper should discuss whether and how consent was obtained from people whose asset is used.
% \item At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
% \end{itemize}
% \item {\bf Crowdsourcing and research with human subjects}
% \item[] Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The paper does not involve crowdsourcing nor research with human subjects.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
% \item According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
% \end{itemize}
% \item {\bf Institutional review board (IRB) approvals or equivalent for research with human subjects}
% \item[] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
% \item[] Answer: \answerNA % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: The paper does not involve crowdsourcing nor research with human subjects.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
% \item Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
% \item We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
% \item For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
% \end{itemize}
% \item {\bf Declaration of LLM usage}
% \item[] Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
% %this research?
% \item[] Answer: \answerNA{} % Replace by \answerYes{}, \answerNo{}, or \answerNA{}.
% \item[] Justification: LLMs were not used as any non-standard, important, or original component of this research.
% \item[] Guidelines:
% \begin{itemize}
% \item The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
% \item Please refer to our LLM policy (\url{https://neurips.cc/Conferences/2025/LLM}) for what should or should not be described.
% \end{itemize}
% \end{enumerate}
\appendix
\input{Appendix-extra}
% \input{NeuRIPS'2025/Appendix-pos-proper}
\input{Appendix-PU-real-scar}
\input{Appendix-PU-real-dist-shift}
\input{Appendix-PU-agnostic}
\end{document}
\usepackage[utf8]{inputenc} % allow utf-8 input
\usepackage[T1]{fontenc} % use 8-bit T1 fonts
\usepackage{hyperref} % hyperlinks
\usepackage{url} % simple URL typesetting
\usepackage{booktabs} % professional-quality tables
\usepackage{amsfonts} % blackboard math symbols
\usepackage{nicefrac} % compact symbols for 1/2, etc.
\usepackage{microtype} % microtypography
\usepackage{xcolor} % colors
\usepackage{graphicx} % Required for inserting images
\usepackage{amsthm}
\usepackage{amsmath,amsfonts,amssymb,mathtools, bbm}
\usepackage[linesnumbered,ruled,vlined]{algorithm2e}
\usepackage{xcolor}
\usepackage{multirow}
\newtheorem{example}{Example}
\newtheorem{theorem}{Theorem}%[subsection]
\newtheorem{lemma}[theorem]{Lemma}
\newtheorem{proposition}[theorem]{Proposition}
\newtheorem{remark}[theorem]{Remark}
\newtheorem{corollary}[theorem]{Corollary}
\newtheorem{conjecture}[theorem]{Conjecture}
\newtheorem{axiom}[theorem]{Axiom}
\newtheorem{disclaimer}[theorem]{Disclaimer}
\newtheorem{assumption}{Assumption}
\newtheorem{defn}{Theorem}
\newtheorem{definition}[defn]{Definition}
\newcommand{\Sandra}[1]{\textcolor{blue}{[{\bf Sandra:} #1]}}
\newcommand{\Shai}[1]{\textcolor{green}{[{\bf Shai:} #1]}}
\newcommand{\FM}[1]{\textcolor{red}{[{\bf Farnam:} #1]}}
\newcommand{\nP}{b}
\newcommand{\nU}{a}
\newcommand{\C}{\mathcal{C}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Y}{\mathcal{Y}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\Nat}{\mathbb{N}}
\newcommand{\tD}{\tilde{\mathcal{D}}}
\newcommand{\cP}{\mathcal{P}}
\newcommand{\cQ}{\mathcal{Q}}
\newcommand{\E}[1]{\mathbb{E}\left[#1\right]}
\newcommand{\cE}[2]{\mathbb{E}_{#1}\left[#2\right]}
\newcommand{\var}[1]{\operatorname{Var}\left[#1\right]}
\newcommand{\VaR}{\operatorname{V@R}}
\newcommand{\argmin}{\operatorname{argmin}}
\newcommand{\VCD}{\operatorname{VCD}}
\newcommand{\CLOS}{\operatorname{CLOS}}
\newcommand{\loss}{L}
\newcommand{\err}{\operatorname{err}}
\newcommand{\terr}{\tilde{\operatorname{err}}}
\newcommand{\rev}{e}
\newcommand{\trev}{\tilde{\rev}}
\newcommand{\clabel}{l}
\newcommand{\freq}{f^U}
\newcommand{\prior}{\alpha}
\newcommand{\claw}{\mathfrak{h}}
\newcommand{\pdist}{\D_{+}}
\newcommand{\ndist}{\D_{-}}
\section{Introduction}
Learning from positive and unlabeled data (PU learning) is a variant of binary classification prediction semi-supervised learning, where the training data consist only of positively labeled and unlabeled examples. PU learning arises in many applications, such as personal advertisement (where a person is labeled according to whether a given add is relevant to them. When a person responds to the add, we know they belong to the set of positive instances. However, we cannot tell the label of unresponsive customers), land cover classification \cite{li2010positive} (say, we wish to classify forest land cover from aerial images, where training data consist of unlabeled land images and forest aerial images), prediction of protein similarity \cite{elkan2008learning} and many other applications like knowledge base completion \cite{bekker2020learning}, disease-gene identification \cite{yang2012positive} and more.
Standard machine learning paradigms, such as empirical risk minimization (namely, training a classifier to minimize the miss-classification loss over the training data) or regularized risk minimization may fail badly in such settings, since their success guarantees rely on having access to labels from both classes (positive and negative labels). We are interested in finite sample size generalization guarantees. Having a weaker supervision than standard fully supervised learning, achieving generalization bounds for PU learning requires stronger assumptions. In this work, we show how some of the common assumptions used in this domain can be relaxed, while also showing some negative, impossibility results.
\textbf{Various setups for PU learning.} We consider the case in which both training samples (the positively labeled and the unlabeled examples) are generated by random processes unknown to the learner.
The learner's goal is to obtain a classifier that minimizes misclassification with respect to \emph{a target evaluation distribution} $\D$ over $\X \times \{0,1\}$ (where $\X$ is the domain set).
% \FM{In this work, we assume the existence of an underlying data-generating distribution $\D$, unknown to the learner, which governs the random process by which both the positively labeled and unlabeled training samples are generated. }
Training data generating setup can be viewed along two basic axes; The first is whether the positively labeled data is generated independently of the unlabeled sample (as opposed to the case where the positively labeled examples are sampled from an already sampled unlabeled set of instances).
%\Shai{The other axis is, given a sample point whose true label is positive, what determines if that label is revealed as part of the training set.}
The other axis is the labeling mechanism through which positive labels are assigned to training examples.
% \FM{My problem is that $\cP$ can be arbitrary and thus negative label can be revealed as positive as well.}
Our paper focuses on scenarios where the unlabeled sample and the positive sample are independent of each other (called \emph{control-set scenarios} by \cite{bekker2020learning}).
%As an example for training-set scenario, consider personal advertisement. For this application, suppose the domain is a set of web-pages, and the positive sample could be the web-pages that are visited or clicked, and the rest of web-pages can be regarded as unlabeled examples \cite{bekker2020learning}.
As an example of a control-set scenario, consider the task of predicting whether a given profile will become a user of a mobile application. For this example, the positive sample can be collected from individuals who are already users of the application, while the unlabeled sample can be drawn from a broader pool of random individuals.
Let $\D_+$ and $\D_-$ denote the conditioning of $\D$ on the label being positive or negative respectively. We consider four setups for how positive training data is generated (See Section~\ref{sec:PU-real-beyond-scar} for formal definitions):
\begin{itemize}
\item \emph{Selected completely at random (SCAR)} \cite{elkan2008learning}: Positive training data is drawn i.i.d. from $\D_+$.
\item \emph{Selected at random (SAR)} \cite{bekker2019beyond}: Positive training data are drawn i.i.d. from a distribution whose support is a subset of the support of $\D_+$.
% referred to as having \emph{sampling bias} \cite{kato2019learning}.
\item \emph{Positive covariate shift (PCS):} Positive training data is drawn from a distribution that shares the same labeling function as $\D$ but has different marginal distributions (referred to as \emph{positive-only shift} in \cite{sakai2019covariate}).
\item \emph{Arbitrary positive distribution shift (APDS):} Positive training data is drawn from an arbitrary distribution (generalization bounds in this case depend on measures of similarity between the two distributions).
\end{itemize}
% \paragraph{Option 2:}
% Our primary focus in this paper is on cases where the positive data is drawn from the projection of $\D$ onto the positively labeled points, also known as being \emph{selected completely at random (SCAR)} \cite{elkan2008learning}. Nevertheless, we also present several results for PU learning beyond the SCAR setup, which we elaborate on in the ``Our Contributions'' section.
% We also study cases where positive data is drawn from a distribution (i) with its support being a subset of the support of $\D$ (called having \emph{sampling bias} \cite{kato2019learning}, or being \emph{selected at random (SAR)} \cite{bekker2019beyond} in the literature) (ii) (called positive only shift in \cite{sakai2019covariate})
Following the common terminology, \emph{realizable} setup refers to learning with respect to data distributions for which some member of the concept class has zero misclassification loss. The setup is \emph{agnostic} PU learning when no such condition is assumed.
%refer to the study of PU learning with a concept class $\C$ that includes a concept $c^* \in \C$ that almost surely does not misclassify any instance as realizable PU learning; otherwise, we refer to it as agnostic PU learning.
The \emph{class prior} is the probability of positive labels, $\alpha := \D[\{(x,1): x \in \X\}]$. In most prior work on PU learning, the class prior is assumed as prior knowledge.
\textbf{Our Contributions.}
The main high-level contributions of this paper are as follows:
\begin{enumerate}
\item We provide finite sample complexity bounds without relying on knowledge of the class prior $\alpha$. To the best of our knowledge, prior provable results in PU learning typically assume that $\alpha$ is known and used by the learner. The only exceptions are \cite{liu2002partially}, which provides a result limited to realizable PU learning under the SCAR setup, and \cite{lee2025learning}, whose results apply specifically to smooth distributions.
% We discussed the differences of their results from ours in the related works section below. \Shai{You should explain the differences here!}
\item We provide new sample complexity upper bounds in a variety of setups, for which such bounds have not been previously proved.
\item We prove novel lower bounds that match existing positive results for the SCAR setup.
\end{enumerate}
In more detail, our contributions are:
% \Shai{The table should clarify very explicitly what results were known and what results are new here. This table does not do it!}
\begin{itemize}
\item \textit{Realizable PU Learning (SCAR setup).} In Theorem~\ref{thm:real-scar-pos-lower-bound}, we provide a lower bound on the sample complexity of positive examples that nearly matches earlier upper bounds (e.g., by \cite{liu2002partially}). Moreover, in Theorem~\ref{thm:real-scar-unlabel-lower-bound}, we also provide a lower bound on the sample complexity of unlabeled examples based on a novel combinatorial parameter that we introduce, called \emph{claw number}.
% In respectively Theorem~\ref{thm:real-scar-pos-lower-bound}, and Theorem~\ref{thm:real-scar-unlabel-lower-bound} we propose the first lower bound for the sample complexity of positive and unlabeled examples of realizable PU learning for the SCAR setup, and our lower bound for positive examples nearly matches existing upper bounds \cite{liu2002partially}.
\item \textit{Realizable PU learning (SAR setup).}
%For this setup, in Theorem~\ref{thm:real-sar-PU}, we show that the algorithm of \cite{liu2002partially} retains finite sample complexity under certain conditions.
We prove the first finite sample complexity for this setup that does not require knowledge of $\alpha$ (Theorem~\ref{thm:real-sar-PU}). We then provide an almost tight lower bound on the sample complexity of positive examples in Theorem~\ref{thm:real-sar-pos-lower-bound}.
% In Theorem~\ref{thm:real-sar-PU} we propose the first algorithm with finite sample complexity for realizable PU learning in the SAR case without para, for which in Theorem~\ref{thm:real-sar-pos-lower-bound} we provide a nearly matching lower bound for sample complexity of positive examples.
\item \textit{Realizable PU learning (PCS setup).}
For this setup, we introduce the first algorithm which guarantees finite sample complexity (Theorem~\ref{thm:real-dist-shift-PU}). We then provide lower bounds on sum of sample complexity of positive and unlabeled examples (Theorem~\ref{thm:real-dist-shift-lower-bound} and Theorem~\ref{thm:real-dist-shift-lower-bound-lip}). These results highlight the differences between the PCS and the SAR setups.
% For this setup, we first derive lower bounds on the sample complexity of PU learning in Theorem~\ref{thm:real-dist-shift-lower-bound} and Theorem~\ref{thm:real-dist-shift-lower-bound-lip}, which highlight the differences between this setup and the SAR setup. Then, in Theorem~\ref{thm:real-dist-shift-PU} we propose the first algorithm with finite sample complexity guarantees for PCS setup.
\item \textit{Agnostic PU Learning (SCAR setup, when $\prior$ is known).} For this setup, in Theorem~\ref{thm:agnostic-PU-lower-bound}, we propose a lower bound on the sample complexity of both positive and unlabeled examples that nearly matches existing upper bounds established in \cite{du2015convex}.
\item \textit{Agnostic PU Learning (SCAR setup, when $\prior$ is unknown).}
While without knowledge of $\prior$ or additional assumptions on the data or concept class it is impossible to find a classifier whose misclassification rate is arbitrarily close to that of the best in the class, we show in Corollary~\ref{cor:PU-agnostic-lag-pos-scar}, that a learner can always find a classifier whose misclassification rate is arbitrarily close to
\(
\frac{\max(\prior, 1 - \prior)}{\min(\prior, 1 - \prior)}
\)
times the misclassification rate of the best concept in the concept class. Moreover, in Corollary~\ref{thm:PU-agnostic-no-alpha-lowerbound}, we show that this multiplicative factor is tight. Our result also yields an improved generalization bound in scenarios where an approximation of $\prior$ is available.
\item \textit{Agnostic PU learning (APDS setup).}
For this setup, we derive the first generalization bounds with finite sample complexity (Theorem~\ref{thm:agnostic-PU-lower-bound}).
\end{itemize}
%In Table~\ref{tab:summary}, we summarize all our contributions, excluding the results presented in Section~\ref{sec:PU-agnostic-upperbound}.
\begin{table}[h] \label{tab:summary}
\centering
\caption{Summary of all results presented in this paper (excluding those in Section~\ref{sec:PU-agnostic-upperbound}). Here, $d$ denotes the VC-dimension of the concept class $\C$ %\cite{doi:10.1137/1116025}
; $k$ is the dimensionality of the input space; $\claw$ is the claw number of $\C$; $r$ is a weight ratio between distribution of positively labeled training data and $\D_+$ ; $\gamma$ is the margin parameter ;$\pi$ is any lower bound on $\alpha$ that is available to the learner. Logarithmic factors are suppressed in this table. }
\small
\begin{tabular}{|c|c c| c|}
\hline
% \textbf{Setting} & \multicolumn{2}{c|}{\textbf{Sample Complexity}} & \textbf{Source}\\
\multicolumn{4}{|c|}{\textbf{Bounds on the sample complexity of PU learning}} \\
\hline
\multirow{2}{*}{Realizable (SCAR)} &
$m^{pos}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{\varepsilon}\right)$
& $m^{unlabel}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{\varepsilon}\right)$
& \cite{liu2002partially} \\
&
$m^{pos}_\C(\varepsilon, \delta) =\tilde{\Omega}\left(\frac{d}{\varepsilon}\right)$ & $m^{unlabel}_\C(\varepsilon, \delta) = \tilde{\Omega}\left(\frac{\claw}{\varepsilon}\right)$
& \textbf{ New results in this work.} \\
% & Theorems~\ref{thm:real-scar-pos-lower-bound} \& \ref{thm:real-scar-unlabel-lower-bound} \\
\hline
\multirow{2}{*}{Realizable (SAR)} &
$m^{pos}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{r\varepsilon}\right)$ & $m^{unlabel}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{\varepsilon}\right)$
% & Theorem~\ref{thm:real-sar-PU} \\
& \textbf{ New results in this work.} \\
& $m^{pos}_\C(\varepsilon, \delta) = \tilde{\Omega}\left(\frac{d}{r\varepsilon}\right)$ & $m^{unlabel}_\C(\varepsilon, \delta) = \tilde{\Omega}\left(\frac{\claw}{\varepsilon}\right)$
% & Theorems~\ref{thm:real-sar-pos-lower-bound} \& \ref{thm:real-scar-unlabel-lower-bound} \\
& \textbf{ New results in this work.} \\
\hline
\multirow{2}{*}{Realizable (PCS)} &
$m^{pos}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{r^2\varepsilon}\right)$& $m^{unlabel}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{\left(\frac{\sqrt{k}}{\gamma}\right)^k + \pi d}{\pi \varepsilon}\right)$
% & Theorem~\ref{thm:real-dist-shift-PU} \\
& \textbf{ New results in this work.} \\
&
\multicolumn{2}{c|}{$m^{pos}_\C(\varepsilon, \delta) + m^{unlabel}_\C(\varepsilon, \delta) = \tilde{\Omega}(1+1 / 2\gamma)^{k/2}$ }
% & Theorem~\ref{thm:real-dist-shift-lower-bound-lip} \\
& \textbf{ New results in this work.} \\
\hline
Agnostic &
$m^{pos}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{\varepsilon^2}\right)$ & $m^{unlabel}_\C(\varepsilon, \delta) = \tilde{O}\left(\frac{d}{\varepsilon^2}\right)$ & \cite{du2015convex}\\
(SCAR, known $\prior$) &
$m^{pos}_\C(\varepsilon, \delta) = \tilde{\Omega}\left(\frac{d}{\varepsilon^2}\right)$& $ m^{unlabel}_\C(\varepsilon, \delta) = \tilde{\Omega}\left(\frac{d}{\varepsilon^2}\right)$
% & Theorem~\ref{thm:agnostic-PU-lower-bound} \\
& \textbf{ New results in this work.} \\
\hline
\end{tabular}
\label{tab:pu-sample-complexity}
\end{table}
% \section{Related works}
% In this section,
\textbf{Related works.} We briefly survey previous theoretical studies of PU learning.
We start with works on the SCAR setup.
% , which is the focus of most of the published literature on PU learning.
% \paragraph{PU Learning in the SCAR setup.}
For the easiest case of realizable learning, \cite{liu2002partially} describe an algorithm with finite sample complexity.
For challenging agnostic PU learning, previously proposed learning rly on a priori knowledge of the class prior (e.g., \cite{du2015convex}).
Consequently, much of research for PU learning for the agnostic SCAR setup is devoted to estimating that prior, $\prior,$. Reliable estimates are only possible with further assumptions on the underlying distribution and concept class.
%\cite{blanchard2010semi}.
These assumptions include (i) \emph{Separability:} non-overlapping support between the $\D_-$ and $\D_+$ \cite{elkan2008learning, du2014class}; (ii) \emph{Anchor set}: requiring a subset of the instance space defined by partial attribute assignment, to be purely positive \cite{scott2015rate, liu2015classification, pmlr-v45-Christoffel15, bekker2018estimating}; (iii) \cite{ramaswamy2016mixture} discuss a generalization of anchor set assumption and call it also separability.
%\FM{@Shai: maybe we should discuss this line too}
(iv) Irreducibility: $\D_-$ cannot be expressed as a linear combination of $\D_+$ and any other distribution \cite{blanchard2010semi, jain2016estimating}.
% However, \cite{blanchard2010semi} shows that without knowledge of $\prior$, it is impossible to find a classifier whose misclassification rate is arbitrarily close to that of the best concept in the class for agnostic PU learning.
Our results do not require any of these assumptions.
% \paragraph{PU Learning beyond SCAR setup.}
Next, we consider studies of PU learning that extend beyond the SCAR setup.
Several articles examine PU learning in the SAR setting \cite{coudray2023risk, dai2023gradpu, gong2021instance, na2020deep, bekker2019beyond, kato2019learning, he2018instance}, among which only \cite{coudray2023risk, gong2021instance, kato2019learning, he2018instance} pursues theoretical analysis (the others focus primarily on empirical evaluations). In contrast to our work, these studies assume that $\prior$ is known. Under that assumption, they provide learnability results applicable to the agnostic PU learning setting.
%Moreover, in contrast to \cite{gong2021instance}, we do not rely on parametric assumptions.
\cite{kato2019learning} focuses on establishing statistical consistency rather than finite sample guarantees. \cite{he2018instance} analyze a special case of the SAR setting, referred to as the \emph{probabilistic gap assumption}.
% \cite{sakai2019covariate} studies linear classifiers under a variant of the PCS setting, where the distribution of the unlabeled training data may also differ from the target evaluation distribution; \cite{hammoudeh2020learning} investigates a setting in which both the positive and unlabeled training data are collected by a distribution that, conditioned on the label being negative, matches the target evaluation distribution. \cite{kumar2023positive} explores several relaxations of the SCAR setup. In all of these works, the theoretical analysis focuses on establishing statistical consistency.
\cite{sakai2019covariate, hammoudeh2020learning, kumar2023positive} discuss statistical consistency for different variations of PU learning. This is in contrast with our focus on finite sample size generalization bounds.
\cite{lee2025learning} studies both the sample complexity and computational complexity in a setting where the unlabeled training data is drawn from a smooth distribution that differs from the target evaluation distribution, while the positive training data is drawn from the target evaluation distribution conditioned on the label being positive.
% \paragraph{Learning from positive examples.}
% The specific case of PU learning in which there is no unlabeled training data is of particular interest. \cite{lee2025learning} provided a characterization of this problem under the realizability assumption and also studied it under the assumption that the target evaluation distribution is smooth. Moreover, \cite{karatapanis2025oracle, lee2024efficient} studied this problem under the assumption that the concept class has bounded \emph{Gaussian Surface Area}.
% This setup has also been studied under the name of the \emph{out-of-distribution detection} problem \cite{NEURIPS2022_f0e91b13, JMLR:v25:23-1257}, where the characterizing conditions have been analyzed for various scenarios.
% which has been studied under various assumptions on the target evaluation distribution and the distribution of positive training data \FM{I know this line is not clear, but I cannot summarize their work in 1-2 line}. \FM{This paragraph is not nice:(}
% There are also a line of work when the positive distribution is known \cite{FJK96, NR09, RG09, AGR13, DDS15, CDS20, JKV23}
% \cite{natarajan1987learning} also studies a similar objective (Further discussed in Section~\ref{sec:pos-learning-proper}).
Note that, due to space constraints, all proofs in this submission are deferred to the appendix.
\section{Setting}
% We consider the following setup for learning with positive and unlabeled examples (PU learning). Let $\X$ be the domain set, and $\Y = \{0, 1\}$ the labels set. Moreover, let $\mathcal{F}_{\X}$ be any $\sigma-$algebras over $\X$ and $\mathcal{F}_{\Y} = 2^\Y$.
% % \FM{I don't even know defining $\sigma-$algebra is necessary or not}
% There are two distributions. A distribution $\D$ over the measurable space $(\X \times \Y, \mathcal{F}_{\X} \otimes \mathcal{F}_{\Y})$, where $\mathcal{F}_{\X} \otimes \mathcal{F}_{\Y}$ is the product of $\sigma-$algebras $\mathcal{F}_{\X}$ and $\mathcal{F}_{\Y}$; and a distribution for \emph{sampling the positively labeled training data} over measurable space $(\X, \mathcal{F}_{\X})$ denoted by $\cP$. Also, denote $\D_{\X}$ to be the marginal distribution of $\D$ over instances. Moreover, given a function $f: \X \to \mathcal Y$ define $\err_{\D}(f) := Pr_{(x, y) \sim \D} [f(x)\neq y]$ as error of $f$ with respect to $\D$.
We consider the following setup for learning with positive and unlabeled examples (PU learning). Let $\X$ be the domain set, $\Y = \{0, 1\}$ the labels set. We consider two distributions, a distribution $\D$ over $\X \times \Y$, and a distribution for \emph{sampling the positively labeled training data} over $\X$ denoted by $\cP$.
Given a function $f: \X \to \mathcal Y$, define $\err_{\D}(f) := Pr_{(x, y) \sim \D} [f(x)\neq y]$ as the error of $f$ with respect to $\D$. Also, define \emph{positive distribution}, and \emph{negative distribution} respectively to be $\D_+(A) := \D(A \mid y = 1)$ and $\D_{-}(A) := \D(A \mid y = 0)$ for every measurable set $A \subseteq \X$. Moreover, denote $\D_{\X}$ to be the marginal distribution of $\D$ over the domain set.
A PU learner takes (i) a sample $S^U$ of size $\nU$ i.i.d. drawn from marginal distribution $\D_{\X}$, and (ii) a sample $S^P$ of size $\nP$ i.i.d. drawn from $\cP$
% we also study cases where $\cP$ can differ from $\D_+$) and
independent of $S^U$, denoted by $S^P$, and similar to classical machine learning, it aims to output a function $f: \X \to \mathcal{Y}$ which minimizes $\err_{\D}(f)$. Formally, a PU learner is a function
$$
\mathcal{A} : \X^* \times \X^* \rightarrow \{0, 1\}^\X.
$$
% We say the learner $\mathcal{A}$ is \emph{proper} if the range of $\mathcal{A}$ is in $\C$. Otherwise, we say $\mathcal{A}$ is \emph{improper}.
% Note that in this setup, we assume that $S^P$ is sampled from $\D_+$. In such cases, we say that the positive data is \emph{selected completely at random}. However, in Sections~\ref{sec:PU-real-beyond-scar} and \ref{sec:PU-agnostic-upperbound}, we also study scenarios where $S^P$ is sampled from a distribution $\cP \neq \D_+$, with the appropriate definitions provided within those sections.
We now establish our framework for evaluating the success of PU learners:
% Note that this setup corresponds to the case-control scenario \cite{bekker2020learning}, where the positive data is \emph{selected completely at random} \cite{}.
% as explained in the introduction.
% \begin{definition}[PU learnability -- SCAR setup] \label{def:PU}
% Let $\C$ be a concept class over domain $\X$, and $\mathcal{G}$ be a set of distributions $\D$ over $\X \times \Y$. We say that concept class $\C$ is PU learnable over the set of distributions $\mathcal{G}$, if there exist functions
% $m^{pos}_{\C}: (0,1) \times (0,1) \to \Nat, m^{unlab}_{\C}: (0,1) \times (0,1) \to \Nat$, and a PU learner $\mathcal{A}$ such that for all $(\varepsilon, \delta) \in (0,1) \times (0,1)$, and distributions $\D \in \mathcal{G}$ if $\nP > m^{pos}_{\C}(\varepsilon, \delta)$ and $\nU > m^{unlab}_{\C}(\varepsilon, \delta)$,
% we have
% $$
% \Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \min_{c \in \C} \err_{\D}(c) + \varepsilon\right] < \delta.
% $$
% We also say $\mathcal{A}$ \emph{PU learns} $\C$ over $\mathcal{G}$.
% \end{definition}
\begin{definition}[PU learnability] \label{def:PU-beyond-scar}
Let $\C$ be a concept class over domain $\X$. Moreover, let $\mathcal{W}$ be a set of pairs $(\D, \cP)$, where $\D$ is a distribution over $\X \times \Y$; and $\cP$ is a distribution over $\X$. We say that concept class $\C$ is PU learnable over the class $\mathcal{W}$ if there exist functions
$m^{pos}_{\C}: (0,1) \times (0,1) \to \Nat, m^{unlab}_{\C}: (0,1) \times (0,1) \to \Nat$, and a PU learner $\mathcal{A}$ such that for all $(\varepsilon, \delta) \in (0,1) \times (0,1)$ and distributions $(\D, \cP) \in \mathcal{W}$ if $\nP > m^{pos}_{\C}(\varepsilon, \delta)$ and $\nU > m^{unlab}_{\C}(\varepsilon, \delta)$,
we have
$$
\Pr_{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \min_{c \in \C} \err_{\D}(c) + \varepsilon\right] < \delta.
$$
We also say $\mathcal{A}$ \emph{PU learns} $\C$ over $\mathcal{W}$.
% when given access to a sample $S^P$ of size $\nP$, generated i.i.d. by $\cP$, and an unlabeled sample $S^U$ of size $\nU$, generated i.i.d. by $\D_{\X}$, with probability at least $1 - \delta$ (over the choice of the samples $S^P$ and $S^U$) $\mathcal{A}$ outputs a function $f$ with $\err_{\D} (f) \leq \inf_{c \in \C} \err_{\D} (c) + \varepsilon$. We also say that
\end{definition}
% We refer to the cases where $\mathcal{G}$ is the set of all distributions $\D$ for which there exists a $c \in \C$ such that it satisfies $\err_{\D} (c) = 0$ as the setup that satisfies the realizability assumption.
% . Also, define the measurable function label probability $\clabel(A) := \D(y = 1 \mid A)$.
% \FM{I think we don't need to define this here}
\textbf{Notations:}
Given any set $J$ and $k \in \mathbb{N}$, let $U_J$ denote the uniform distribution over $J$, and define $J^k := \left\{ (j_1, \ldots, j_k) \mid j_1, \ldots, j_k \in J \right\}$, and $[k] := \{1, \ldots, k\}$. Given a family of distributions $\D_\omega$ over $\X \times \{0, 1\}$, where $\omega$ ranges over some parameter set, we respectively denote the marginal over $\X$, the positive distribution, and the negative distribution of $\D_\omega$ by $\D_{\X, \omega}$, $\D_{+, \omega}$, and $\D_{-, \omega}$.
Let $\C$ be a concept class over $\X$. Define $\C \Delta \C := \{ c \oplus c' \mid c, c' \in \C \}$. Moreover, denote the best classifier as \(c^\star := \arg\min_{c \in \C} \err_\D(c),\) and $\min_{c \in \C} \err_\D(c)$ as the \emph{approximation error}. Furthermore, for a function $c: \X \rightarrow \{0, 1\}$ and distribution $\D$ over $\X \times \{0, 1\}$, define the \emph{false positive rate} as $\err_{\D}^+(c) := \Pr_{x \sim \D_+}[c(x) \neq 1]$, and the \emph{false negative rate} as $\err_{\D}^-(c) := \Pr_{x \sim \D_-}[c(x) \neq 0]$. Given a subset $B \subseteq \X$, define $\C \cap B := \{ c \cap B \mid c \in \C \}$. Moreover, for a multiset $S = (x_1, x_2, \dots, x_m) \in \X^*$, define $\text{Domain}(S) := \{x \mid x \in S\}$. Define the \emph{restriction of $S$ to $B$} denoted by $S \mid B$ as the subsequence of elements $x_i \in S$ such that $x_i \in B$.
\section{Analysis of Realizable PU Learning --SCAR setup} \label{sec:PU-real-scar}
In this section, we study PU learning under the realizability assumption in the SCAR setup. It is already known that every concept class with finite VC dimension is PU learnable in this setting \cite{liu2002partially}. We begin by establishing lower bounds on the sample complexity. In particular, we provide a lower bound on the sample complexity of positive examples that nearly matches the upper bound established by \cite{liu2002partially}.
\begin{restatable}{theorem}{thmscarposlow}
\label{thm:real-scar-pos-lower-bound}
Let $\C$ be a concept class with VC dimension $d \geq 2$ over the domain $\X$. There exists a $M > 1$ such that for $\nP \leq M \left( \frac{d + \ln(1 / \delta)}{\varepsilon} \right)$ and every $\nU \in \Nat$, $(\varepsilon, \delta) \in (0, 1) \times (0, 1)$, and PU learner $\mathcal{A}$ there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ such that
$
\Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D}( \mathcal{A}(S^P, S^U) \geq \varepsilon\right] > \delta.
$
\end{restatable}
% Proof appears in the appendix.
Next, we provide a lower bound for the sample complexity of unlabeled samples with respect to a combinatorial parameter we call \emph{claw number}. Claw number is formally defined in the following. As mentioned in Remark~\ref{remark:claw-num}, claw number is always smaller than VC dimension.
\begin{definition}
Let $\C$ be a concept class over domain $\X$. We define \emph{claw number} of $\C$ to be the largest $\claw \in \Nat$ such that for every $m \geq \claw$, there exists a $B \subseteq \X$ with $|B| = m$ such that $\{ O \subseteq B \mid |O| = m - \claw \} \subseteq \C \mid B $. If no such $\claw$ exists, we say the claw number of $\C$ is $0$.
\end{definition}
\begin{remark} \label{remark:claw-num}
Claw number of a class is always less than or equal to VC dimension. This is because for every $B \subseteq \X$ with $|B| \geq 2\claw$ we have $\VCD(\{ O \subseteq B \mid |O| = |B| - \claw \}) \geq \claw$.
\end{remark}
% \begin{restatable}{theorem}{thmscarunlabelNFL}
% \label{thm:real-scar-unlabel-nFL}
% Let $\C$ be a concept class over $\X$. Let $\claw \geq \Nat$, $\nP \in \Nat$ and
% $\nU \leq \claw$. Suppose for a $m \geq 2\nP + 4 \claw$, there exists a $|B| = m$ such that $\{ O \subseteq B \mid |O| = m - 2 \claw \} \subseteq \C \mid B $. Then for every PU learner $\mathcal{A}$ there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ such that
% $
% \Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D}( \mathcal{A}(S^P, S^U) \geq \frac{1}{8}\right] > \frac{1}{16}.
% $
% \end{restatable}
\begin{restatable}{theorem}{thmscarunlow}
\label{thm:real-scar-unlabel-lower-bound}
Let $\C$ be a concept class with claw number $\claw \geq 1$. There exists a $M > 1$ such that for $\nU \leq M \left( \frac{\claw + \ln(1 / \delta)}{\varepsilon} \right)$ and every $\nP \in \Nat$, $(\varepsilon, \delta) \in (0, 1) \times (0, 1)$, and PU learner $\mathcal{A}$ there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ such that
$
\Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D}( \mathcal{A}(S^P, S^U) \geq \varepsilon\right] > \delta.
$
\end{restatable}
Note that \cite{lee2025learning} showed that no concept class $\C$ with $\VCD(\C_\cap) = \infty$ ($\C_\cap$ is defined bellow) is realizably PU learnable in the SCAR setup without access to unlabeled examples. The following proposition shows that Theorem~\ref{thm:real-scar-unlabel-lower-bound} recovers this result.
\begin{restatable}{proposition}{propcintclaw}
For a concept class $\C$, let $\C_{\cap} := \left\{ \bigcap_{c \in A} c \mid \text{finite } A \subseteq \C \right\}$. Then $\VCD(\C_{\cap}) = \infty$ if and only if the claw number of $\C$ is at least 1.
\end{restatable}
% when the positive data are selected completely at random (positive data is sampled from $\D_+$).
% in the SCAR setup.
% \cite{liu2002partially} proved that every concept class with finite VC dimension is PU learnable over the class of distributions realized by $\C$.
We then restate the Theorem 1 of \cite{liu2002partially} in Corollary~\ref{cor:real-scar-PERM}, providing an alternative proof based on the notion of $\varepsilon$-nets, which we formally define below. Our proof also leads to new results for the SAR and PCS setups, presented in Section~\ref{sec:PU-real-beyond-scar}.
\begin{definition} [$\varepsilon-$net]
Let $\X$ be some domain, $\mathcal{B} \subseteq 2^{\X}$ a collection of subsets of $\X$ and $\mathcal{Q}_\X$ a distribution over $\X$. An $\varepsilon$-net for $\mathcal{W}$ with respect to $\mathcal{Q}_\X$ is a subset $N \subseteq \X$ that intersects every member of $\mathcal{B}$ that has $\mathcal{Q}_\X$-weight at least $\varepsilon$.
\end{definition}
Let us also elaborate on the learning algorithm \cite{liu2002partially} introduced, appearing in \eqref{eq:PERM}. Note that \eqref{eq:PERM} simply selects the concept with the fewest number of 1s over $S^U$ among all concepts consistent with $S^P$. In this sense, it can be seen as a counterpart to \emph{empirical risk minimization} in the PU learning setting. We therefore refer to any concept returned by \eqref{eq:PERM} as a \emph{positive empirical risk minimizer} (PERM).
\begin{equation} \label{eq:PERM}
\operatorname{argmin}_{c \in \C, Domain(S^P) \subseteq c}
\left|c \mid S^U \right|
\end{equation}
% Lemma~\ref{lem:epnet-pi} can be extracted from the proof of Theorem 1 of \cite{liu2002partially}, which uses the notion of epsilon nets, formally defined in the following. Lemma~\ref{lem:epnet-pi} will assist us heavily in Section~\ref{sec:pos-learning}, and we left it's proof in the appendix.
% The following lemma will immediately result in Corollary~\ref{cor:real-scar-PERM}.
\begin{restatable}{lemma}{lemepnet}
\label{lem:epnet-pi}
Let $\C$ be a realizable concept class with VC dimension $d$ over domain $\X$.
Let
% $c^\star$ be any concept with $\err_{\D}(c^\star) = 0$, and
$S$ be a sample i.i.d. drawn from $\D_{\X}$ and $T \in \X^*$ be an $\varepsilon-$net for $\C \triangle \C$ on $\D_+$ such that $Domain(T) \subseteq c^\star$. Denote $c^{PU} := \argmin_{c \in \C, Domain(T) \subseteq c} |c \mid S|$. Then there exists a $M > 1$ such that if $|S| > M \left(\frac{d \ln (1 / \varepsilon) + \ln(1 / \delta)}{\varepsilon} \right)$, then with probability $1 - 2\delta$ we have $\err_{\D}(c^{PU}) \leq 14 \varepsilon$.
\end{restatable}
% Notice that a very similar result to Theorem 1 of \cite{liu2002partially} would be a direct implication of Lemma~\ref{lem:epnet-pi}.
\begin{restatable} {corollary} {corscarup}
[Theorem 1 of \cite{liu2002partially}]\label{cor:real-scar-PERM}
Let $\C$ be a concept class with VC dimension $d$ over the domain $\X$. Let $\mathcal{W}$ be a set of duos $(\D, \D_+)$ such that $\D$ is realized by $\C$. Then $\C$ is PU learnable over $\mathcal{W}$ with sample complexity $m^{pos}_\C(\varepsilon, \delta), m^{unlabel}_\C(\varepsilon, \delta) = O \left( \frac{d \ln (1 / \varepsilon) + \ln( 1/ \delta)}{\varepsilon} \right)$.
\end{restatable}
\begin{proof}
For a fixed constant $M$, as long as $\nP > M \left(\frac{\VCD(\C \triangle \C) \ln (1 / \varepsilon) + \ln(1 / \delta)}{\varepsilon} \right)$ we have that $S^P$ with probability $1 - \delta$ is an $\varepsilon-$net for $\C \triangle \C$ on $\D_+$ (e.g., see \cite{hausslerwelzl1987}). Since $\VCD(\C \triangle \C) \leq 2\VCD(\C) + 1$ (it can be shown similar to the manner claim 1 of \cite{ben1998combinatorial} was proved), combining this with Lemma~\ref{lem:epnet-pi} completes the proof.
\end{proof}
% Proof of this theorem appears in the appendix.
\section{Analysis of Realizable PU Learning --Beyond SCAR}
\label{sec:PU-real-beyond-scar}
In this section, we study PU learning under the realizability assumption when positive examples are sampled from a distribution $\cP$ which can differ from $\D_+$.
Throughout this section we consider distributions $\D$ with deterministic labels, i.e., $\D(y = 1 \mid x)$ is always zero or one for every $x \in \X$, and we define $\clabel(x) := \D(y = 1 \mid x)$ to be the \emph{labeling function}.
We study two classes of distributions for sampling positive examples $\cP$:
% We study SAR and PCS setup
(i) Selected at random (SAR): For any distribution $\rev$ over $\X$, define $\D_\rev(A) = \int \D_+(A) d \rev$, and $\cP$ belongs to
$$
\begin{aligned}
\mathcal{K}^{sar}_{\D} := \left \{ \D_\rev \mid \text{any distribution
} e \text{ over } \X \right \}.
\end{aligned}
$$
% Which in the literature is known by positive sample being selected at random (SAR).
% To provide an intuition on why we are interested in this case, you could also assume that $S^P$ is collected in the following manner. Assume that there is a sample $S_1 = ((x_1, y_1), (x_2, y_2), ..., (x_m, y_m))$ i.i.d. drawn $\D$ and independent of $S^U$. Let $i_1, ..., i_{k}$ be the index in $[m]$ where $y_{i_j} = 1$ for $j \in [k]$, and denote $L^P = (x_{i_1}, ..., x_{i_k})$. Then each $x \in S^P$ is drawn from $L^P$ in proportionate with probability $\rev(x)$, where we call $\rev(x)$ \emph{individual propensity score function}.
(ii) Positive covariate shift (PCS): $\cP$ belongs to
$$
\begin{aligned}
\mathcal{K}^{cov}_{\D} := \left \{ \cP \mid \cP(A) = 0 \text{ if } \D_+(A) = 0 \text{ and } \D(A) > 0, \text{ $A$ is measurable set} \right \}
\end{aligned}
$$
% We refer to this case as having positive covariate shift.
Note that the condition $\cP \in \mathcal{K}_{D}^{sar}$ is equivalent to having $\cP(A) = 0$ when $\D_+(A) = 0$ for any measurable set $A$, i.e., support of $\cP$ being a subset of the support of $\D_+$. Thus, $\mathcal{K}^{cov}_{\D}$ is a generalization of the previous case.
% \end{enumerate}
% For this purpose, we generalize our framework for measuring the success of PU learning we defined in Definition~\ref{def:PU} as:
% \begin{definition}[PU learnability --general definition] \label{def:PU-beyond-scar}
% Let $\C$ be a concept class over domain $\X$. Moreover, let $\mathcal{W}$ be a set of pairs $(\D, \cP)$, where $\D$ is a distribution over $\X \times \Y$; and $\cP$ is a distribution over $\X$. We say that concept class $\C$ is PU learnable over the class $\mathcal{W}$ if there exist functions
% $m^{pos}_{\C}: (0,1) \times (0,1) \to \Nat, m^{unlab}_{\C}: (0,1) \times (0,1) \to \Nat$, and a PU learner $\mathcal{A}$ such that for all $(\varepsilon, \delta) \in (0,1) \times (0,1)$ and distributions $(\D, \cP) \in \mathcal{W}$ if $\nP > m^{pos}_{\C}(\varepsilon, \delta)$ and $\nU > m^{unlab}_{\C}(\varepsilon, \delta)$,
% we have
% $$
% \Pr_{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \min_{c \in \C} \err_{\D}(c) + \varepsilon\right] < \delta.
% $$
% We also say $\mathcal{A}$ \emph{PU learns} $\C$ over $\mathcal{W}$.
% % when given access to a sample $S^P$ of size $\nP$, generated i.i.d. by $\cP$, and an unlabeled sample $S^U$ of size $\nU$, generated i.i.d. by $\D_{\X}$, with probability at least $1 - \delta$ (over the choice of the samples $S^P$ and $S^U$) $\mathcal{A}$ outputs a function $f$ with $\err_{\D} (f) \leq \inf_{c \in \C} \err_{\D} (c) + \varepsilon$. We also say that
% \end{definition}
% Consider $\mathcal{K}^{det}$ to be the set of distributions $\D$ such that the label of instances are deterministic, i.e., for every $x \in \X$ we have $\D(y = 1 \mid x)$
% % \FM{If I want to define it properly, I should use Radon-Nikodym}
% is always zero and or 1. For $\D \in \mathcal{K}^{det}$ we define $\clabel(x) := \D(y = 1 \mid x)$ to be the \emph{labeling function}.
% % \FM{Again here there is confusion}.
% Our main results in this section, are for $\D \in \mathcal{K}^{det}$.
We begin by analyzing the simpler case where $\cP \in \mathcal{K}^{sar}_{\D}$, and then extend our results to the more general setting $\mathcal{K}^{cov}_{\D}$. Even when $\cP$ belongs to $\mathcal{K}^{sar}_{\D}$, additional assumptions on $\cP$ are required for the PU learning problem to be well-posed. For example, consider the case where $\cP$ is a single point mass on a positively labeled instance. In this scenario, the PU learner would only observe one labeled example, rendering the learning task trivial and unsolvable. To avoid such cases, we impose a common assumption when dealing with distribution shift: a bounded \emph{weight ratio} between $\cP$ and $\D_+$ \cite{}. The weight ratio is formally defined as follows.
\begin{definition} [weight ratio]
Let $\mathcal{B} \subseteq 2^{X}$ be a collection of subsets of the domain $\X$ measurable with respect to both $\cQ_{\X, 1}$ and $\cQ_{\X, 2}$. We define the weight ratio of the source distribution and the target distribution with respect to $\mathcal{B}$ as
$$
R_{\mathcal{B}}\left(\cQ_{\X, 1}, \cQ_{\X, 2}\right)=\inf_{\substack{A \in \mathcal{B}(\X) \\ \cQ_{\X, 2}(A) \neq 0}} \frac{\cQ_{\X, 1}(A)}{\cQ_{\X, 2}(A)},
$$
We denote the weight ratio with respect to the collection of all sets that are $\cQ_1$ and $\cQ_2$-measurable by $R\left(\cQ_1, \cQ_2\right)$.
\end{definition}
Our sample complexity upper bound for the case where $\cP \in \mathcal{K}^{sar}_{\D}$ is
the direct implication of Lemma~\ref{lem:ben2012-lem3} proven by \cite{ben2012hardness}, which we state below
\begin{lemma}[Lemma 3 of \cite{ben2012hardness}] \label{lem:ben2012-lem3}Let $\mathcal{X}$ be some domain, $\mathcal{B} \subseteq 2^{\mathcal{X}}$ a collection of subsets of $\mathcal{X}$, and $\mathcal{Q}_{\X, 1}$ and $\mathcal{Q}_{\X, 2}$ distributions over $\mathcal{X}$ with $R:=R_{\mathcal{B}}\left(\mathcal{Q}_{\X, 1}, \mathcal{Q}_{\X, 2}\right) \geq 0$. Then every $R\varepsilon$-net for $\mathcal{B}$ with respect to $\mathcal{Q}_{\X, 1}$ is an $\varepsilon$-net for $\mathcal{B}$ w.r.t. $\mathcal{Q}_{\X, 2}$.
\end{lemma}
\begin{restatable}{theorem}{thmsarup} \label{thm:real-sar-PU}
Let $\C$ be a concept class over domain $\X$ with VC dimension $d$ and $r \in (0, 1)$. Let $\mathcal{W}$ be a set of duos $(\cP, \D)$ such that $\D$ is realized by $\C$, $\cP \in \mathcal{K}^{sar}_{\D}$, and $R_{\C \Delta \C}\left(\cP, \D_+ \right) \geq r$. Then PERM algorithm \eqref{eq:PERM} PU learns $\C$ over $\mathcal{W}$ with sample complexity $m^{unlabel}_\C(\varepsilon, \delta) = O\left(\frac{d \ln (1 / \varepsilon) + \ln(1 / \delta)}{\varepsilon} \right)$ and $m^{pos}_\C(\varepsilon, \delta) = O\left(\frac{d \ln (1 / r \varepsilon) + \ln(1 / \delta)}{r \varepsilon} \right)$.
\end{restatable}
Next we derive a nearly tight lower bound for the sample complexity of positive examples when $\cP \in \mathcal{K}^{sar}_{\D}$ and $R\left(\cP, \D_+ \right) \geq r$.
\begin{restatable}{theorem}{thmsarposlow}
\label{thm:real-sar-pos-lower-bound}
Let $\C$ be a concept class over domain $\X$ with VC dimension $d \geq 2$ and $r \in (0, 1)$. There exists a $M > 1$ such that for $\nP \leq M \left( \frac{d + \ln(1 / \delta)}{r \varepsilon} \right)$ and every $\nU \in \Nat$, $\varepsilon, \delta \in (0, 1) \times (0, 1)$, and PU learner $\mathcal{A}$, there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ and a distribution $\cP \in \mathcal{K}^{sar}_{\D}$ such that $R\left(\cP, \D_+ \right) \geq r$ and
$
\Pr_{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D}( \mathcal{A}(S^P, S^U) \geq \varepsilon\right] > \delta.
$
\end{restatable}
The proof of Theorem~\ref{thm:real-sar-pos-lower-bound} closely follows that of Theorem~\ref{thm:real-scar-pos-lower-bound}
% (additional details deferred to the appendix).
(see appendix).
Now, we can shift our focus to $\cP \in \mathcal{K}^{cov}_{\D}$. In Theorem~\ref{thm:real-dist-shift-lower-bound}, inspired by \cite{ben2012hardness} we show that for $\cP \in \mathcal{K}^{cov}_\D$ no weight ratio assumption is sufficient for PU learnability, unless the total number of positive and unlabeled samples depends on the size of the domain. Therefore, similar to \cite{ben2012hardness} in the cases where $\cP \in \mathcal{K}^{cov}_{\D}$, in addition to a weight ratio assumption, we assume that the labeling function $\clabel$ is a \emph{$\gamma-$margin classifier w.r.t. $\D$}, and $\D$ is \emph{realizable by $\C$ with margin $\gamma$}. These notions are formally defined in the following. Moreover, we also assume that $\D[\{(x,1): x \in \X\}]$ has a constant lower bound (note that we are not assuming $\D[\{(x,1): x \in \X\}]$ is known).
% The proof of Theorem~\ref{thm:real-sar-pos-lower-bound} closely follows that of Theorem~\ref{thm:real-scar-pos-lower-bound}, but we provide additional details in the appendix. Now, we shift our focus to the case where $\cP \in \mathcal{K}^{\text{cov}}_{\D}$. In Theorem~\ref{thm:real-dist-shift-lower-bound}, inspired by \cite{ben2012hardness}, we show that for $\cP \in \mathcal{K}^{\text{cov}}_\D$, no weight ratio assumption is sufficient for PU learnability unless the total number of positive and unlabeled samples depends on the size of the domain. Therefore, in the cases where $\cP \in \mathcal{K}^{\text{cov}}_{\D}$, in addition to the weight ratio condition, we assume that the labeling function $\clabel$ is a \emph{$\gamma$-margin classifier w.r.t. $\D_\X$} (formally defined below), as was also assumed by \cite{ben2012hardness}; and that $\D(y = 1)$ has a constant lower bound (note that we are not assuming $\D(y = 1)$ is known).
% $\D$ is \emph{realizable by $\C$ with margin $\gamma$}
% Moreover, we also assume that $\D(y = 1) \geq \pi$ for some constant $\pi$.
\begin{definition} [realizable with $\gamma-$margin]
Let $\X \subseteq \mathbb{R}^k, \D$ be a distribution over $\X \times \{0, 1\}$ and $c: \X \rightarrow\{0,1\}$ a classifier. For all $x \in \X$, denote $B_\gamma(x)$ as the norm-2 ball with radius $\gamma$ centered on $x$. We say that $c$ is a $\gamma$-margin classifier with respect to $\D_\X$ if for all $x \in \X$ whenever $\D_\X\left(B_\gamma(x)\right)>0$ then $c(y)=c(z)$ holds for all $y, z \in B_\gamma(x)$.
We say that a class $\C$ realizes $\D$ with margin $\gamma$ if the optimal (zero-error) classifier $c^\star$ is a $\gamma$-margin classifier.
\end{definition}
Note that a function $c$ being a $\gamma$-margin classifier with respect to $\D_\X$ is equivalent to $c$ satisfying the Lipschitz property with Lipschitz constant $1/2\gamma$ on the support of $\D_\X$.
\begin{restatable}{theorem}{thmcovlow}
\label{thm:real-dist-shift-lower-bound}
Consider any finite domain $\X$. There exists a concept class $\C_{0, 1}$ with $\VCD(\C_{0, 1}) = 1$, such that for every PU learner $\mathcal{A}$, and $\varepsilon$ and $\delta$ with $2\varepsilon+\delta<1 / 2$, $\nP, \nU \in \Nat$ such that $\nP+\nU < \sqrt{\frac{2(1-2(2\varepsilon+\delta))|\X|}{3}} - 2$, there exists a distribution $\D$ over $\X \times \{0, 1\}$ with deterministic labels which is realized by $\C_{0, 1}$ and $\cP \in \mathcal{K}^{cov}_{\D}$ where $R(\cP, \D_+) = 1/ 2$,
% , $d_{\C_{0, 1} \triangle \C_{0, 1}}(\cP, \D_+) = 0$
$\D[\{(x,1): x \in \X\}]\geq 1 / 2$ and
$
\Pr_{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D}( \mathcal{A}(S^P, S^U) \geq \varepsilon\right] > \delta.
$
% such that for every $\varepsilon$ and $\delta$ with $2\varepsilon+\delta<1 / 4$ and $\nP, \nU \geq 216$, no algorithm can $(\varepsilon, \delta, \nP, \nU)$-solve the PU learning problem for the class $\mathcal{W}$ of duos $\left(\cP, \D \right)$
% with $\D \in \mathcal{K}^{det}$ and $\cP \in \mathcal{K}^{cov}_{\D}$,
% and $C(\cP, \D_+) = 1/ 2$,
% $d_{\C_{0, 1}, \C_{0, 1}}(\cP, \D_+) = 0$,
% and $\min_{c \in \C_{0, 1}} \err_{\D}(c) = 0$ if $\nP+\nU < \sqrt{\frac{(1-4(2\varepsilon+\delta))|\X|}{27}} - 2$.
\end{restatable}
% Proof of this theorem appears in the appendix.
The following theorem is inspired by Theorem 2 of \cite{ben2012hardness}, which establishes a lower bound on sample size for infinite domains under the additional assumptions that the labeling function is $\lambda$-Lipschitz and that $\D[\{(x,1): x \in \X\}] \geq \frac{1}{2}$. As shown, even with these additional assumptions, the total number of samples must be at least exponential in the Lipschitz constant. This can be viewed as the additional cost incurred when $\cP \in \mathcal{K}^{\text{cov}}_{\D}$.
\begin{restatable}{theorem}{thmcovlowlip}
\label{thm:real-dist-shift-lower-bound-lip}
Let $\X = [0,1]^k$. There exists a concept class $\C_{0, 1}$ with $\VCD(\C_{0, 1}) = 1$, such that for every PU learner $\mathcal{A}$, and $\varepsilon$ and $\delta$ with $2\varepsilon+\delta<1 / 2$, $\nP, \nU \in \Nat$ such that $\nP+\nU < \sqrt{\frac{2(1 + \lambda)^k(1-2(2\varepsilon+\delta))}{3}} - 2$, there exists a distribution $\D$ over $\X \times \{0, 1\}$ with deterministic labels which is realized by $\C_{0, 1}$ and $\cP \in \mathcal{K}^{cov}_{\D}$ where $C(\cP, \D_+) = 1/ 2$,
% $d_{\C_{0, 1} \triangle \C_{0, 1}}(\cP, \D_+) = 0$,
$\D[\{(x,1): x \in \X\}] \geq 1 / 2$ and
$\clabel$ is a $\lambda$-Lipschitz labeling function and
$
\Pr_{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D}( \mathcal{A}(S^P, S^U) \geq \varepsilon\right] > \delta.
$
% There exists a concept class $\C_{0, 1}$ with $\VCD(\C_{0, 1}) = 1$, such that $\varepsilon > 0$ and $\delta > 0$ be such that $2\varepsilon + \delta < 1/4$ and $\nP, \nU \geq 216$, let $\lambda > 1$ and let $\mathcal{W}_{\lambda}$ be the set of duos $(\cP, \D)$ such that $\D \in \mathcal{K}^{det}$ and $\cP \in \mathcal{K}^{cov}_{\D}$
% and $C(\cP, \D_+) = 1/ 2$,
% $d_{\C_{0, 1}, \C_{0, 1}}(\cP, \D_+) = 0$,
% and $\min_{c \in \C_{0, 1}} \err_{\D}(c) = 0$ and $\clabel$ is $\lambda$-Lipschitz labeling functions. Then no PU learner can $(\nP, \nU, \varepsilon, \delta)$-solve the PU learning problem for the class $\mathcal{W}_{\lambda}$ unless $\nP+\nU < \sqrt{\frac{(1 + \lambda)^d(1-4(2\varepsilon+\delta))}{27}} - 2$.
\end{restatable}
Next, we present Algorithm~\ref{alg:real-dist-shift}, designed for the case where $\cP \in \mathcal{K}^{cov}_\D$. The algorithm is inspired by the domain adaptation method introduced in \cite{ben2012hardness}. In the standard domain adaptation setting, the goal is to minimize the error with respect to a target distribution $\cQ_T$, given labeled samples from a source distribution $\cQ_S$ and unlabeled samples from $\cQ_T$.
Algorithm~\ref{alg:real-dist-shift} adapts this approach to the PU learning setting, with $\cQ_S = \cP$ and $\cQ_T = \D_+$ (with labels being 1). However, unlike domain adaptation, PU learning lacks access to unlabeled samples from $\D_+$; instead, it only has access to unlabeled samples from $\D_\X$. To account for this difference, two key modifications are made to the algorithm from \cite{ben2012hardness}:
(i) Instead of using a sample $T$ from $\D_+$, Algorithm~\ref{alg:real-dist-shift} uses the unlabeled sample $S^U$ drawn from $\D_\X$;
(ii) The algorithm outputs a PERM
% (Positive Empirical Risk Minimizer)
rather than an ERM.
% (Empirical Risk Minimizer).
Notice that in Theorem~\ref{thm:real-dist-shift-PU}, we require the number of unlabeled samples to be exponential with respect to $1/\gamma$ (as it was required for the total number of samples to be exponential with respect to the Lipschitz constant in our lower bound appearing in Theorem~\ref{thm:real-dist-shift-lower-bound-lip}). However, in many learning scenarios, unlabeled data is abundantly available while labeled data is difficult to obtain, which makes this algorithm more practically appealing.
\begin{algorithm}\label{alg:real-dist-shift}
\caption{Algorithm for PU learning in the positive covariate shift setup}
\KwIn{$S^P$ i.i.d. sampled from $\cP$ with label 1 and an unlabeled i.i.d. sample $S^U$ from $\D_{\X}$ and a margin parameter $\gamma$.}
Partition the domain $[0,1]^k$ into a collection $\mathcal{B}$ of boxes (axis-aligned rectangles) with sidelength $(\gamma / \sqrt{k})$ \;
Obtain sample $S^{\prime}$ by removing every point in $S^P$, which is sitting in a box that is not hit by $S^U$ \;
\Return{ $\operatorname{argmin}_{c \in \C, Domain(S') \subseteq c} \left|c \mid S^U \right|$}
\end{algorithm}
\begin{restatable}{theorem}{thmcovup}
\label{thm:real-dist-shift-PU}
Let $\X=[0,1]^k, \gamma > 0$ a margin parameter, $\pi, r>0$ and $\C$ be a realizable concept class with VC dimension $d < \infty$. Let $\mathcal{W}$ to be the set of duos $(\cP, \D)$ such that:
(i) $\cP \in \mathcal{K}^{cov}_{\D}$, (ii) $\D$ is realizable by $\C$ with margin $\gamma$ and has deterministic labels, (iii) $\D[\{(x,1): x \in \X\}] \geq \pi$, (iv) The labeling function $\clabel$ is a $\gamma$-margin classifier with respect to $\D_\X$, (v) $R_{\mathcal{I}}\left(\cP, \D_+ \right) \geq r$ for the class $\mathcal{I}=(\C \Delta \C) \sqcap \mathcal{B}$, where $\mathcal{B}$ is a partition of $[0,1]^k$ into boxes of sidelength $\gamma / \sqrt{k}$.
% \begin{itemize}
% \item $\cP \in \mathcal{K}^{cov}_{\D}$, $\D$ is realizable by $\C$ with margin $\gamma$ and has deterministic labels, and $\D (y = 1) \geq \pi$.
% \item The labeling function $\clabel$ is a $\gamma$-margin classifier with respect to $\D_\X$.
% \item $R_{\mathcal{I}}\left(\cP, \D_+ \right) \geq r$ for the class $\mathcal{I}=(\C \Delta \C) \sqcap \mathcal{B}$, where $\mathcal{B}$ is a partition of $[0,1]^k$ into boxes of sidelength $\gamma / \sqrt{k}$.
% % \item
% \end{itemize}
Then Algorithm~\ref{alg:real-dist-shift} PU learns $\C$ over $\mathcal{W}$ with sample complexity
% $$m^{pos}_\C(\varepsilon, \delta) = M \left(\frac{d+\ln (1 / \delta)}{r^2(1-\varepsilon)^2 \varepsilon} \ln \left(\frac{d}{r (1-\varepsilon) \varepsilon}\right)\right),$$
% and
\begin{gather*}
m^{pos}_\C(\varepsilon, \delta) = O \left(\frac{d \ln \left(1 / (r (1-\varepsilon) \varepsilon)\right)+\ln (1 / \delta)}{r^2(1-\varepsilon)^2 \varepsilon} \right), \\
m^{unlabel}_\C(\varepsilon, \delta) = O \left(\frac{(\sqrt{ k} / \gamma)^k \ln \left((\sqrt{ k} / \gamma)^k / \delta\right) }{\pi \varepsilon } + \frac{d \ln(1 / \varepsilon) +\ln (1 / \delta)}{\varepsilon} \right).
\end{gather*}
\end{restatable}
% Proof of this theorem appears in the appendix.
\section{Analysis of the Agnostic PU Learning} \label{sec:PU-agnostic}
In this section we analyze agnostic PU learning. It is already known that with the knowledge of class prior $\prior$, every class with finite VC dimension is PU learnable \cite{du2015convex} in the SCAR setup. In Section~\ref{sec:PU-agnostic-lower-bound} we derive a nearly matching lower bound on both the sample complexity of unlabeled examples and positive examples to \cite{du2015convex} upper bounds. Then, we show that for a concept class $\C$ with more than two concepts, without the knowledge of $\prior$, no PU learner--without access to $\prior$--can achieve an error less than $\frac{\max(\prior, 1 - \prior)}{\min(\prior, 1 - \prior)}$ times the approximation error even in the SCAR setup (which makes the PU learning task impossible). Furthermore, in Section~\ref{sec:PU-agnostic-upperbound}, we complement this result by showing that for every concept class, there exists an algorithm whose error is arbitrarily close to $\frac{\max(\prior, 1 - \prior)}{\min(\prior, 1 - \prior)}$ times the approximation error in the SCAR setup. Finally, we derive generalization bounds for settings where $S^P$ is drawn from an arbitrary distribution $\cP$.
%. Moreover,
% \begin{example} \label{ex:prior-needed}
% Let $\X = [0, 1]$ and $\C$ be class of threshold functions, and $\D = U_{\X} \times Ber(\prior)$, where $U_{\X}$ is the uniform distribution and $Ber(\prior)$ be bernouli distribution with parameter $\prior$, and $\cP = \D_+$. Consider any PU learner $\mathcal A$. Given some $r \in [0, 1]$, for both $\prior = r$ and $\prior = 1 - r$ we have $\min_{c \in \C} \err_{\D}(c) = \min(r, 1-r)$. However, either for $\prior = r$ or $\prior = 1 - r$ we have $\E{\err_{\D} (\mathcal A(S^P, S^U))} \geq \frac{1}{2}$.
% \end{example}
% \begin{proof}
% We have $\err_{\D}(c) := Pr(c(x) = 1) * (1 - \prior) + Pr(c(x) = 0) \prior = \prior + Pr(c(x) = 1) (1 - 2\prior)$. This directly implies the first part of our claim. Next we derive
% $$
% \begin{aligned}
% & \E{\err_{\D} (\mathcal A(S^P, S^U))} = \prior
% + (1 - 2\prior) \E{\mathcal A(S^P, S^U) (x) = 1}
% \end{aligned}
% $$
% Note that both $\D_+$ and $\D^N_{\X}$ are uniform distributions. Therefore, estimating ratio of positive labels $\prior$ is impossible using positive and unlabelled data. Therefore, $\mathcal{A}$ should be independent of $\prior$. Moreover, notice that irrespective of $\E{\mathcal A(S^P, S^U) (x) = 1}$ either we have $r + (1 - 2r) \E{\mathcal A(S^P, S^U) (x) = 1} \geq \frac{1}{2}$ or $1 - r + (2r - 1) \E{\mathcal A(S^P, S^U) (x) = 1} \geq \frac{1}{2}$. This completes the proof.
% \end{proof}
% Although Corollary~\ref{thm:PU-agnostic-no-alpha-lowerbound} shows that without the realizability assumption or knowledge of $\alpha$ almost no concept class is PU learnable,
\subsection{Lower Bounds --SCAR setup} \label{sec:PU-agnostic-lower-bound}
The following theorem provides an almost tight lower bound on the sample complexity of both positive and unlabeled examples, assuming the learner knows that $\alpha = \frac{1}{2}$. We prove this theorem by reducing it to a problem called \emph{the generalized weighted die problem}, which is a problem inspired by \cite{ben2011learning}.
Detailed Proof of the theorem is deferred to the appendix.
\begin{restatable}{theorem}{thmagnolow}
\label{thm:agnostic-PU-lower-bound}
Let $\C$ be a concept class with $\VCD(\C) = d$ where $d \geq 4$. Consider $\mathcal{W}$ to be the set of duos $(\D, \D_+)$ with $\D[\{(x,1): x \in \X\}] = 0.5$. Then $\C$ is PU learnable over $\mathcal{W}$ with sample complexity $m^{unlabel}_\C(\varepsilon, \delta), m^{pos}_\C(\varepsilon, \delta) = \Omega\left( \frac{d + \ln( 1/ \delta)}{\varepsilon^2}\right)$ and $m^{unlabel}_\C(\varepsilon, \delta), m^{pos}_\C(\varepsilon, \delta) = O\left( \frac{d \ln(1 / \varepsilon) + \ln( 1/ \delta)}{\varepsilon^2}\right)$.
\end{restatable}
Next, we present a lower bound on the generalization of PU learners for the cases where $\alpha$ is unknown.
\begin{theorem} \label{thm:PU-agnostic-no-alpha-lowerbound}
Let $\C$ be a concept class over $\X$ containing at least two distinct concepts. Then, for every $\eta \in (0, 1)$, $\nP, \nU \in \Nat$, and PU learner $\mathcal{A}$, there exists a distribution $\D$ over $\X \times \{0, 1\}$ with $\alpha \in \{\eta, 1 - \eta\}$, where $\alpha := \D[\{(x,1) : x \in \X\}]$, such that
\[
\Pr_{S^P \sim \D_{+}^{\nP},\, S^U \sim \D_{\X}^{\nU}} \left[
\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \frac{\max(\prior, 1 - \prior)}{\min(\prior, 1 - \prior)} \min_{c \in \C} \err_\D(c)
\right] = 1.
\]
\end{theorem}
\begin{proof}
Let $x$ be any instance such that two concepts in $\C$ disagree on its label. Define distribution $\D_0$ over $\X \times \{0, 1\}$ to assign probability $\eta$ on $(x, 1)$ and $1 - \eta$ over $(x, 0)$, and $\D_1 := 1 - \D_0$. Then for any $z \in \{0, 1\}$ we have $\min_{c \in \C} \err_{\D_z}(c) = \min(\eta, 1 - \eta)$ and $\D_{+, z} = \D_{\X, z} = \mathbbm{1}_{\{x\}}$. Thus, for any $\nP, \nU \in \Nat$ and $S^P \sim \D_{+, z}^{\nP}, S^U \sim \D_{\X, z}^{\nU}$ there exists a $z \in \{0, 1\}$ such that $\err_{\D_z} \left( \mathcal A( S^P, S^U) \right) = \max( \eta, 1 - \eta)$. This completes the proof.
\end{proof}
\begin{remark}
Our poof also demonstrates that, even when the approximation error is known, almost no concept class is PU learnable. This finding is particularly noteworthy because agnostic PU learning with known approximation error can be viewed as a relaxation of the realizable PU learning setting.
\end{remark}
\subsection{Upper bounds}
\label{sec:PU-agnostic-upperbound}
% The algorithm we study
% The algorithm we propose for agnostic
We start by proposing an algorithm for agnostic PU learning that, for a given $\gamma > 0$, outputs a concept which minimizes the \emph{Lagrangian PU empirical loss} $\hat{\err}^\gamma : \mathcal{C} \rightarrow \mathbb{R}^{\geq 0}$, defined as
\begin{equation} \label{eq:lagrangian-err}
\hat{\err}^\gamma(c) := \frac{\left|c \mid S^U\right|}{\nU} + \gamma \cdot \frac{\nP - \left|c \mid S^P\right|}{\nP}.
\end{equation}
Note that the PERM algorithm minimizes $\frac{\left|c \mid S^U\right|}{\nU}$ while assuming that the empirical error of $c$ (for realizable concept classes) is zero on $S^P$. Since $\frac{\nP - \left|c \mid S^P\right|}{\nP}$ is the empirical error on $S^P$, this algorithm can also be viewed as a Lagrangian function for the PERM algorithm. Also, notice that when $\gamma = 2 \alpha$, $\hat \err^\gamma$ will be equivalent to the surrogate loss introduced in \cite{du2015convex} when the loss function is the zero-one loss. We begin by analyzing the SCAR setup.
% We start by proposing Algorithm~\ref{alg:lag-pos} for agnostic PU learning. Moreover, note that the PERM algorithm minimizes $\frac{\left|c \mid S^U\right|}{\nU}$ while assuming that the empirical error of $c$ (for realizable concept classes) is zero on $S^P$. Since $\frac{\nP - \left|c \mid S^P\right|}{\nP}$ is the empirical error on $S^P$, Algorithm~\ref{alg:lag-pos} can also be viewed as a Lagrangian function for the PERM algorithm. We begin by analyzing the SCAR setup. Also, notice that when $\gamma = 2 \alpha$, $\hat \err^\gamma$ will be equivalent to the surrogate loss introduced in \cite{du2015convex} when the loss function is the zero-one loss.
% \begin{algorithm}\label{alg:lag-pos}
% \caption{Lagrangian Loss ERM}
% \KwIn{$S^P$ i.i.d. sampled from $\cP$ an unlabeled i.i.d. sample $S^U$ from $\D_{\X}$ and a Lagrangian parameter $\gamma > 0$.}
% % \KwOut{A concept $c^{PU}$}
% \BlankLine
% For any $c \in C$ define $\hat \err^\gamma(c) := \frac{\left|c \mid S^U\right|}{\nU} + \gamma \frac{\nP - \left|c \mid S^P\right|}{\nP} $ \;
% \Return{$c^{PU} = \argmin_{c \in \C} \hat \err^\gamma (c)$.}
% \end{algorithm}
\begin{restatable}{theorem}{thmagnoscarup}
\label{thm:PU-agnostic-lag-pos-scar}
Let $\C$ be any concept class over domain $\X$ with VC dimension $d$, and let $\cP = \D_+$. Given any $\gamma \geq \prior$, denote
% $c^{PU}$ be the output of Algorithm~\ref{alg:lag-pos}.
$c^{PU} = \argmin_{c \in \C} \hat \err^\gamma (c)$.
There exists $M > 1$ such that for all $c \in \C$, if $|S^P|, |S^U| > \frac{M (d + \ln (1 / \delta))}{ \varepsilon^2}$, then with probability $1 - 4 \delta$ we have
$$
\begin{aligned}
\err_{\D}(c^{PU}) \leq
\max\left(\frac{\gamma - \prior}{\prior} , \frac{\prior}{\gamma - \prior} \right)\left( \err_\D(c) + 2(1 + \gamma) \varepsilon \right)
\end{aligned}
$$
\end{restatable}
% Proof of this theorem appears in the appendix.
\begin{remark}
Let's also suppose as a prior knowledge we have access to $\hat \prior \approx \prior$ where $2\hat \prior \geq \prior$. Then one can incorporate the prior knowledge by setting $\gamma = 2 \hat \prior$.
% \FM{One could think of a Bayesian approach too}
In particular, if $\prior$ was known with $\gamma = 2\prior$, we would have $\err_{\D}(c^{PU}) \leq \err_{\D}(c) + 6 \varepsilon$. This is consistent with \cite{du2015convex} results for cases where the class prior is known.
\end{remark}
The following corollary is a direct consequence of applying Theorem~\ref{thm:PU-agnostic-lag-pos-scar} with $\gamma = 1$.
\begin{corollary} \label{cor:PU-agnostic-lag-pos-scar}
For any concept class $\C$ with VC dimension $d$, there exists a PU learner $\mathcal{A}$ and a constant $M > 1$ such that for every $\alpha, \varepsilon, \delta \in (0,1)$ and for all $\nP, \nU > \frac{M (d + \ln (1 / \delta))}{\varepsilon^2}$, and for any distribution $\D$ over $\X \times \{0, 1\}$ with $\D[\{(x,1): x \in \X\}]= \alpha$, the following holds: for any sample $S^P$ of size $\nP$ drawn i.i.d. from $\D_+$ and any sample $S^U$ of size $\nU$ drawn i.i.d. from $\D_{\X}$, with probability at least $1 - \delta$,
$$
\err_{\D} \left( \mathcal A( S^P, S^U) \right) \leq \frac{\max(\prior, 1 - \prior)}{\min(\prior, 1 - \prior)} \left(\min_{c \in \C} \err_\D(c) + 4 \varepsilon \right).
$$
\end{corollary}
% \subsection{Arbitrary positively labeled training data distribution} \label{sec:PU-agnostic-arb}
Finally, we examine the most general PU learning setting. We derive generalization bounds that hold for arbitrary concept classes and any distribution $\cP$ by combining Theorem~\ref{thm:PU-agnostic-lag-pos-scar} with \cite{ben2010theory} results. These bounds involve the $\C \Delta \C$ distance, which we formally define below.
% Note that in this section in contrast with Section~\ref{sec:PU-real}, we do not have any separate results for $\cP \in \mathcal{K}^{sar}_{\D}$ or $\cP \in \mathcal{K}^{cov}_{\D}$.
\begin{definition} \cite{kifer2004detecting}
Given a domain $\X$ and a collection $\mathcal{B}$ of subsets of $\X$, let $\cQ_{\X, 1}, \cQ_{\X, 2}$ be probability distributions over $\X$, such that every set in $\mathcal{B}$ is measurable with respect to both distributions. The $\mathcal{B}$-distance between such distributions is defined as
$$
d_{\mathcal{B}}\left(\cQ_{\X, 1}, \cQ_{\X, 2}\right)=2 \sup _{B \in \mathcal{B}}\left|\Pr_{\cQ_{\X, 1}}[B]-\Pr_{\cQ_{\X, 2}}[B]\right|
$$
\end{definition}
% The following Theorem is a
\begin{restatable}{theorem}{thmagnoarbup}
\label{thm:PU-agnostic-lag-pos-arb}
Let $\C$ be any concept class over domain $\X$ with VC dimension $d$, and let $\cP$ be any arbitrary distribution. Given any $\gamma \geq \prior$, denote
% $c^{PU}$ be the output of Algorithm~\ref{alg:lag-pos}.
$c^{PU} = \argmin_{c \in \C} \hat \err^\gamma (c)$.
There exists $M > 1$ such that for all $c \in \C$, if $|S^P|, |S^U| > \frac{M (d + \ln (1 / \delta))}{ \varepsilon^2}$, then with probability $1 - 4 \delta$ we have
$$
\begin{aligned}
\err_{\D}(c^{PU}) \leq \max\left(\frac{\gamma - \prior}{\prior} , \frac{\prior}{\gamma - \prior} \right)\left(err_{\D}(c) + 2(1 + \gamma) \varepsilon +
2 \gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right) \right)
\end{aligned}
$$
Where
$\lambda^P := \min_{c \in \C} \left( \err_{\D}^+(c) + \err_{\cP}(c, 1)\right)$ and $\err_{\cP}(c, 1) := \Pr_{x \sim \cP}(c(x) \neq 1)$.
\end{restatable}
% Proof of this theorem appears in the appendix.
\section{Useful Theorems}
\begin{lemma} \label{lem:highprob-to-expect}
Let $Z$ be a random variable such that $Z \in [0, \gamma]$ and $\Pr[ Z > \varepsilon] \leq \delta$. Then
$
\E{Z} < \gamma \delta + \varepsilon (1 - \delta).
$
\end{lemma}
\begin{lemma}[Multiplicative Chernoff bounds \cite{motwani1996randomized}] \label{lem:mult-chern-bound} Let $X_1, \ldots, X_m$ be independent random variables drawn according to some distribution $\mathcal{D}$ with mean $p$ and support included in $[0,1]$. Then, for any $\gamma \in\left[0, \frac{1}{p}-1\right]$, the following inequality holds for $\widehat{p}=\frac{1}{m} \sum_{i=1}^m X_i$:
$$
\begin{aligned}
& \mathbb{P}[\widehat{p} \geq(1+\gamma) p] \leq e^{-\frac{m p \gamma^2}{3}} \\
& \mathbb{P}[\widehat{p} \leq(1-\gamma) p] \leq e^{-\frac{m p \gamma^2}{2}}
\end{aligned}
$$
\end{lemma}
\begin{theorem}[Hoeffding Inequality \cite{hoeffding1994probability}] \label{thm:Hoeffding}
Let $X_1, \ldots, X_n$ be independent random variables such that $a_i \leq X_i \leq b_i$ almost surely. Consider the sum of variables,
$$
S_n=X_1+\cdots+X_n
$$
Then Hoeffding's theorem states that, for all $t>0$,
$$
\begin{gathered}
\mathrm{Pr}\left(\left|S_n-\mathrm{E}\left[S_n\right]\right| \geq t\right) \leq 2 \exp \left(-\frac{2 t^2}{\sum_{i=1}^n\left(b_i-a_i\right)^2}\right)
\end{gathered}
$$
\end{theorem}
\begin{theorem}[Chebyshev's inequality \cite{feller1991introduction}] \label{thm:chebyshev}
Let $X$ be a random variable with bounded non-zero variance. Then for any $k>0$,
$$
\operatorname{Pr}(|X-\E{X}| \geq k ) \leq \frac{\operatorname{Var}[X]}{k^2}
$$
\end{theorem}
\begin{lemma} [Slud's inequality \cite{slud1977distribution}] \label{lem:slud}
For $S \sim \operatorname{Bin}(m, p)$ where $p \leq \frac{1}{2}$ and $b$ is an integer with $m p \leq b \leq m(1-p)$ then
$$
P[S \geq b] \geq P\left[Z \geq \frac{b-m p}{\sqrt{m p(1-p)}}\right]
$$
where $Z \sim N(0,1)$ is a normally distributed random variable with mean of 0 and standard deviation of 1 .
\end{lemma}
\begin{lemma} [Normal tail bound \cite{tate1953double}] \label{lem:norm-tail-bound}
For standard Gaussian random variable $Z \sim N(0,1)$ and $x \geq 0$ we have
$$
P[Z \geq x] \geq \frac{1}{2}\left(1-\sqrt{1-\mathrm{e}^{-x^2}}\right)
$$
\end{lemma}
\section{Missing Proofs from Section~\ref{sec:PU-real-scar}}
% \subsection{Proof of Lemma~\ref{lem:epnet-pi}}
% \FM{todo: make the arguments more concrete}
% Before proceeding with the proof of Lemma~\ref{lem:epnet-pi}, we begin by presenting Corollary 5 from \cite{liu2002partially}.
% \subsection{Proof of Theorem~\ref{thm:real-scar-pos-lower-bound}}
\thmscarposlow*
\begin{proof}
\textit{Proof of $m^{pos}_{\C}(\varepsilon, \delta) = \Omega(\frac{d}{\varepsilon})$.} We prove that for $d \geq 9$ we have $m^{pos}_{\C}(\varepsilon, \frac{1}{2000}) \geq \frac{d - 1}{32000 \varepsilon}$. Let $B = \{x_1, ..., x_d\}$ be a set of size $d$ shattered by $\C$, and $\varepsilon = \min \left(\frac{d - 1}{32000 \nP}, 0.0005\right)$ and $\rho = 2000 \varepsilon$. Denote $\bar B := B \setminus \{x_d\}$, and for any $O \subseteq \bar B$ define $\D_O$ over $X \times \{0, 1\}$ be
\begin{equation} \label{eq:real-scar-pos-lower-bound-dists}
\D_O(\{(x, y)\}) := \begin{cases}
1 - \rho & x = x_d \text{ ,and } y = 1 \\
\frac{\rho}{d - 1} & x \in O, \text{ and } y = 1 \\
\frac{\rho}{d - 1} & x \notin O, \text{ and } y = 0 \\
0 & \text{o.w.}
\end{cases}
\end{equation}
Define $\mathcal{W}^{scar-pos}_{\rho, d} := \{ (\D_O, \D_{+, O}) \mid O \subseteq \bar B\}$. Note that for proving the claim it is enough to show that for every PU learner $\mathcal A$, there exists a $O^\star \subseteq \bar B$ such that
$$
\Pr_{S^P \sim \D_{+, O^\star}^{\nP}, S^U \sim \D_{\X, O^\star}^{\nU}} \left[\err_{\D_{O^\star}} \left( \mathcal{A}(S^P, S^U) \right) \geq \varepsilon \right] > \frac{1}{500}.
$$
Note that, for all $O, O' \subseteq B$ we have $\D_{\X, O} = \D_{\X, O'}$. Therefore, for this set of distributions, the unlabeled sample does not help the learner, and it doesn't affect the proof. Thus, for the sake of simplicity, we shorten $\mathcal{A}(S^P, S^U)$ to $\mathcal{A}(S^P)$. Also, without loss of generality, we can assume that $\mathcal A$ always predicts 1 on instance $x_d$.
From this point on for any sample $S$ with $Domain(S) = B$, denote $\bar S := S \setminus \{x_d\}$. Next, define event $E$ to be the event that $|O| \geq \frac{d - 1}{4}$ and $|\bar{S^P}| \leq \frac{d - 1}{8}$. Since maximum is no less than the average, we have
\begin{equation} \label{eq:real-scar-pos-lower-bound-1}
\begin{aligned}
& \max_{O \subseteq \bar B} \cE{S^P \sim \D_{+, O}^{\nP}}{\err_{\D_O}\left(\mathcal{A}(S^P)\right)} \\
& \quad \geq \cE{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}}{\err_{\D_O}\left(\mathcal{A}(S^P)\right)} \\
& \quad \geq \cE{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}}{ \err_{\D_O}\left(\mathcal{A}(S^P)\right) \mid E} \Pr_{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}} [E]
% & = \frac{1}{2^{d - 1}} \sum_{O \in \bar B} \sum_{S^P \in O \cup ^{\nP}} \cE{S^P \sim \D_{+, O}^{\nP}}{\err_{\D_O}\left(\mathcal{A}(S^P)\right)} \\
% \\
\end{aligned}
\end{equation}
We first derive a lower bound for $\Pr_{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}} [E]$. First, note that since $O \sim U_{2^{\bar B}}$ we have $|O| \sim Bin(d - 1, \frac{1}{2})$. Thus, using the Multiplicative Chernoff bound, as long as $d \geq 9$
\begin{equation} \label{eq:real-scar-pos-lower-bound-2}
Pr_{O \sim U_{2^{\bar B}}} \left[|O| < \frac{d - 1}{4} \right] \geq \left(1 - \exp\left(-\frac{d - 1}{16} \right) \right) > 0.3
\end{equation}
Next fix any $O \subseteq \bar B$ with $|O| \geq \frac{d - 1}{4}$. For every $i$ such that $x_i \in O$, and $j \in [\nP]$ let the random variable $Y_{i, j}$ be 1 if the $j$th sample drawn from $\D_{+, O}$ is $x_i$ and 0 otherwise. Note that $Y_{i, j}$ is simply a Bernoulli with parameter at least $\frac{\rho}{d - 1}$. Then, $|\bar{S^P}| = \sum_{i : x_i \in O} \sum_{j = 1}^{\nP} Y_{i, j}$. Therefore, using the Multiplicative Chernoff bound (Lemma~\ref{lem:mult-chern-bound}) for any $\gamma \in \left[0, \frac{ (d - 1)}{|O|} - 1\right]$ we have
\begin{equation*}
\Pr_{S \sim \tilde \D_{+, O}^{\nP}} \left[|\bar{S^P} | \geq (1 + \gamma) \nP \rho \right] \leq e^{-\frac{\nP \rho \gamma^2 |O|}{3 (d - 1)}} \leq e^{-\frac{\nP \rho \gamma^2 }{12}}
\end{equation*}
Set $\gamma = 1$. Note that $\rho \leq \frac{d - 1}{16 \nP}$. Thus
\begin{equation} \label{eq:real-scar-pos-lower-bound-3}
\Pr_{S \sim \tilde \D_{+, O}^{\nP}} \left[|\bar{S^P} | > \frac{d - 1}{8} \right] \leq e^{-\frac{d - 1}{192}} < 0.96
\end{equation}
Combining \eqref{eq:real-scar-pos-lower-bound-2} and \eqref{eq:real-scar-pos-lower-bound-3} we derive that $\Pr_{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}} [E] > 0.012$.
Next we try to derive a lower bound for $\cE{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}}{ \err_{\D_O}\left(\mathcal{A}(S^P)\right) \mid E}$. Note that for a given $S^P$, due to symmetry for all $O, O' \subseteq \bar B$ such that $Domain(\bar{S^P}) \subseteq O, O'$ and $|O| = |O'|$ we have
$$
\Pr_{S \sim \D_{+, O}^{\nP}}[S = S^P] = \Pr_{S \sim \D_{+, O'}^{\nP}}[S = S^P]
$$
Moreover, it is clear that for $O, O' \subseteq \bar B$ such that $Domain(\bar{S^P}) \subseteq O, O'$ and $|O| \geq |O'|$ we also have
$$
\Pr_{S \sim \D_{+, O}^{\nP}}[S = S^P] \leq \Pr_{S \sim \D_{+, O'}^{\nP}}[S = S^P]
$$
Next fix any $S^P$ with $|\bar{S^P}| \leq \frac{d - 1}{8}$. Since given fix $S^P$, smaller $O$ are more likely, due to symmetry we can conclude that given event $E$, for every $x \in \bar B \setminus S^P$ the probability of $x \in O$ is at most $\frac{1}{2}$. Moreover, note that for all $|O| \geq \frac{d - 1}{4}$ we have
$$
\frac{|O \setminus S^P|}{|\bar B \setminus S^P|} \geq \frac{\frac{d - 1}{8}}{d - 1} = \frac{1}{8}.
$$
Thus, again due to symmetry, given event $E$ for every $x \in \bar B \setminus S^P$ the probability of $x \in O$ is at least $\frac{1}{8}$. Thus, no matter what is $\mathcal{A}$ we have
$$
\cE{O \sim U_{2^{\bar B}}, S^P \sim \D_{+, O}^{\nP}}{ \err_{\D_O}\left(\mathcal{A}(S^P)\right) \mid E} \geq \frac{|\bar B \setminus S^P| \rho }{8 (d - 1)} \geq \frac{7}{64} \rho
$$
Combining this with \eqref{eq:real-scar-pos-lower-bound-1}, we conclude that there exists a $O^\star \subseteq \bar B$
$$
\cE{S^P \sim \D_{+, O^\star}^{\nP}}{\err_{\D_{O^\star}}\left(\mathcal{A}(S^P)\right)} > \frac{7 * 0.012 }{64}\rho > 0.001 \rho
$$
Note that since $\mathcal{A}$ always predicts the label of $x_d$ to be $1$, its error is always less than $\rho$. Therefore, using Lemma~\ref{lem:highprob-to-expect} we derive
$$
\begin{aligned}
\Pr_{S^P \sim \D_{+, O^\star}^{\nP}}[\err_{\D_{O^\star}}\left(\mathcal{A}(S^P)\right) \geq \varepsilon] &> \frac{0.001 \rho - \varepsilon }{ ( \rho - \varepsilon)} \\
& = \frac{\varepsilon}{ 2000 \varepsilon - \varepsilon } \\
& > \frac{1}{2000}
\end{aligned}
$$
\textit{Proof of $m^{pos}_{\C}(\varepsilon, \delta) \geq \Omega \left(\frac{\ln(1 / \delta)}{\varepsilon} \right)$.} Next, we prove that for every $\nU \in \Nat$, $\varepsilon< \frac{1}{2}$, $\delta \in (0, 1)$, $\nP = \frac{\ln \left(\frac{1}{2 \delta}\right)}{2 \varepsilon}$ and PU learner $\mathcal{A}$ there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ such that
$$
\Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \varepsilon \right] > \delta.
$$
Let $B = \{x_1, x_2\} \subseteq \X$ be any set shattered by $\C$. For $z \in \{0, 1\}$ and $(x, y) \in \X \times \{0, 1\}$ define $\D_z$ over $\X \times \{0, 1\}$ as
$$
\D_z(\{(x, y)\}) = \begin{cases}
\varepsilon & x = x_1 \text{ and } y = z\\
1 - \varepsilon & x = x_2 \text{ and } y = 1 \\
0 & \text{o.w.}
\end{cases}
$$
Note that both $\D_0$ and $\D_1$ are realized by $\C$. First we try to derive a lower bound for the probability that the sample $S^P \sim \D_{+, 1}^{\nP}$ doesn't contain $x_1$. Note that, it is easy to see that the function $f(a) = \frac{-\ln(1 - a)}{a}$ always has a positive derivative for all $a < 1$. Therefore, for all $\varepsilon < \frac{1}{2}$ we have
$$
\frac{-\ln(1 - \varepsilon)}{\varepsilon} < 2 \ln(2) < 2
$$
Thus,
\begin{equation} \label{eq:real-scar-pos-lower-bound-4}
\begin{aligned}
\Pr_{S^P \sim \D_{+, 1}^{\nP}}[x_1 \notin S^P] &= (1 - \varepsilon)^{\nP} = e^{-\varepsilon \frac{-\ln(1 - \varepsilon)}{\varepsilon} \nP} > e^{ - 2\varepsilon \nP} = 2 \delta.
\end{aligned}
\end{equation}
Thus since $\Pr_{S^P \sim \D_{+, 0}^{\nP}}[x_1 \notin S^P] = 1$. We derive
\begin{equation}
\begin{aligned}
& \min_{z \in \{0, 1\}} \Pr_{S^P \sim \D_{+, z}^{\nP}, S^U \sim \D_{\X,z}^{\nU}} \left[\err_{D_z}(\mathcal{A}(S^P, S^U)) \geq \varepsilon \right] \\
& \stackrel{(i)}{\geq} \min_{z \in \{0, 1\}} \Pr_{S^P \sim \D_{+, z}^{\nP}, S^U \sim \D_{\X,z}^{\nU}} \left[ \mathcal{A}(S^P, S^U)(x_1) \neq z \right]\\
& \geq \min_{z \in \{0, 1\}} \Pr_{S^P \sim \D_{+, z}^{\nP}}[S^P = \{x_2\}^{\nP}] \Pr_{S^U \sim \D_{\X,z}^{\nU}} \left[\mathcal{A}(\{x_2\}^{\nP}, S^U)(x_1) \neq z \right] \\
& > 2 \delta \min_{z \in \{0, 1\}} \Pr_{S^U \sim \D_{\X,z}^{\nU}} \left[ \mathcal{A}(\{x_2\}^{\nP}, S^U)(x_1) \neq z \right] \\
& \stackrel{(ii)}{=} 2 \delta \min \left ( \Pr_{S^U \sim \D_{\X, 0}^{\nU}} \left[ \mathcal{A}(\{x_2\}^{\nP}, S^U)(x_1) = 0 \right], 1 - \Pr_{S^U \sim \D_{\X, 0}^{\nU}} \left[ \mathcal{A}(\{x_2\}^{\nP}, S^U)(x_1) = 0 \right] \right) \\
& \geq \delta
\end{aligned}
\end{equation}
Where (i) is due to the fact that whenever the learner makes a mistake at $x_1$ the error will be at least $\varepsilon$, and (ii) is due to the fact that $\D_{\X, 0} = \D_{\X, 1}$.
\end{proof}
% \subsection{Proof of Theorem~\ref{thm:real-scar-unlabel-lower-bound}}
\thmscarunlow*
\begin{proof}
\textit{Proof of $m^{unlabel}_\C(\varepsilon, \delta) = \Omega \left( \frac{\claw}{\varepsilon} \right)$.} First we prove that for every $\nP, \nU \in \Nat$, and PU learner $\mathcal{A}$ there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ such that
$$
\Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \min \left[\frac{7}{2400}, \frac{7 \claw}{4800 \nU}\right] \right] > \frac{1}{1000}.
$$
Let $\varepsilon := \min \left[\frac{7}{2400}, \frac{7 \claw}{4800 \nU}\right]$, $\rho := \frac{1200 \varepsilon}{7}$, and $m := \max \left( 2(\nP + \nU), \claw \nP, \frac{\claw}{\rho} \right) + \claw$. Let $B$ be any set such that $\{ O \subseteq B \mid |O| = m - \claw \} \subseteq \C$. For any $O \subseteq B$ with $|O| = \claw$ and any $(x, y) \in \X \times \{0, 1\}$ we define the distribution $\D_O$ over $\X \times \{0, 1\}$ as
\begin{equation}
\D_O(\{(x, y)\}) := \begin{cases} \rho / \claw & \text{if } x \in O \text{, and } y = 0 \\
(1 - \rho) / (m - \claw) & \text{if } x \in B \setminus O \text{, and } y = 1 \\
0 & \text{o.w. }
\end{cases}
\end{equation}
Note that all such $\D_O$ are realized by $\C$. We prove our claim holds for one of $\D_O$, where $O \in V$.
% Moreover, since $m \geq \frac{\claw}{\rho} + \claw$, we have $\frac{(1 - \rho)}{m - \claw} \leq \frac{(1 - \rho) \rho}{\claw} < \frac{\rho}{\claw}$.
Next, suppose a learner predicts more than $\claw + \frac{(m - \claw) \rho}{1 - \rho}$ instances of $B$ to be negative. Then, since exactly $\claw$ instances of $B$ are negative in every distribution, the learner will predict more than $\frac{(m - \claw) \rho}{1 - \rho}$ positive instances of $B$ to be negative. Therefore, regardless of the distribution, its error will be more than $\rho$, which is greater than the error of always predicting positive. Thus, without loss of generality, we can assume that the learner always predicts fewer than $\claw + \frac{(m - \claw) \rho}{1 - \rho}$ negative labels. In the worst case, all of the negative predictions are over positive instances, in which case the error will be less than
$$
\claw \cdot \frac{\rho}{\claw} + \left(\claw + \frac{(m - \claw) \rho}{(1 - \rho)}\right) \frac{1 - \rho}{m - \claw} \leq \rho + 2\frac{(m - \claw) \rho}{(1 - \rho)} \cdot \frac{1 - \rho}{m - \claw} = 3 \rho
$$
where the inequality is due to $m \geq \frac{\claw}{\rho} + \claw$. In conclusion, without loss of generality, we can assume that any learner always incurs an error less than $3\rho$.
Define $V = \{ O \subseteq B \mid |O| = \claw\}$. Note that $|V| = \left(\begin{array}{c}m\\ \claw\end{array}\right)$. Since maximum is no less than the average, we derive
\begin{equation} \label{eq:unlabel-sam-lower-bound-1}
\begin{aligned}
& \max_{O \in V} \cE{S^P \sim \D_{+, O}^{\nP}, S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
&
\geq \frac{1}{\left(\begin{array}{c}m\\ \claw\end{array}\right)} \sum_{O \in V} \cE{S^P \sim \D_{+, O}^{\nP}, S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
& = \frac{1}{\left(\begin{array}{c}m\\ \claw\end{array}\right) (m - \claw)^{\nP}} \sum_{O \in V} \sum_{S^P \in (B \setminus O)^{\nP}} \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
& > \frac{\left(\begin{array}{c}m-\nP\\ \claw\end{array}\right)}{\left(\begin{array}{c}m\\ \claw\end{array}\right)}\frac{1}{ \left(\begin{array}{c}m-\nP\\ \claw\end{array}\right) m^{\nP}} \sum_{O \in V} \sum_{S^P \in (B \setminus O)^\nP} \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
& > \frac{1}{3 \left(\begin{array}{c}m-\nP\\ \claw\end{array}\right) m^{\nP}} \sum_{O \in V} \sum_{S^P \in (B \setminus O)^\nP} \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
% & =\frac{1}{3 \left(\begin{array}{c}m-\claw\\ \claw\end{array}\right) m^{\nP}} \sum_{S^P \in O^{\nP}} \sum_{O \in V \setminus S^P} \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}
% \geq \cE{O \sim U_V}{\cE{S^P \sim \D_{+, O}^{\nP}, S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}} \\
% & = \cE{O \sim U_V}{\cE{S^P \sim U_{(B \setminus O)^{\nP}}}{ \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}}} \\
% & = \cE{S^P \sim U_{B^{\nP}}}{\cE{O \sim U_{V \setminus S^P}}{ \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}}} \\
\end{aligned}
\end{equation}
Where the last line is due to the fact that since $m \geq \claw(\nP) + \claw$, we have
$$
\frac{\left(\begin{array}{c}m-\nP\\ \claw\end{array}\right)}{\left(\begin{array}{c}m\\ \claw\end{array}\right) } > \left(\frac{m - \nP - \claw}{m - \claw}\right)^\claw
= \left( \frac{\claw - 1}{\claw} \right)^\claw \geq \frac{1}{e}
$$
Next, for any $S^P \in B^{\nP}$ define $W(S^P) := \{ O \subseteq B \setminus S^P \mid |O| = \claw\}$, and notice that since $ |B \setminus S^P| \geq m - \nP$ we have $|W(S^P)| \geq \left(\begin{array}{c}m-\nP\\ \claw\end{array}\right)$. Therefore, using the fact that the average is no less than the minimum, we derive
\begin{equation} \label{eq:unlabel-sam-lower-bound-2}
\begin{aligned}
& \frac{1}{3 \left(\begin{array}{c}m-\nP\\ \claw\end{array}\right) m^{\nP}} \sum_{O \in V} \sum_{S^P \in (B \setminus O)^{\nP}} \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
& =\frac{1}{3 \left(\begin{array}{c}m-\nP\\ \claw\end{array}\right) m^{\nP}} \sum_{S^P \in B^{\nP}} \sum_{O \in W(S^P)} \cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
& \geq \frac{1}{3} \min_{S^P \in B^{\nP}} \cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}}
% & \geq \frac{1}{3 \left(\begin{array}{c}m-\claw\\ \claw\end{array}\right) m^{\nP}} \sum_{S^P \in O^{\nP}} \sum_{O \in W(S^P)} \left( \mathbb{E}_{S^U \sim \D_{\X, O}^{\nU}} \left[ \frac{\rho}{\claw} \sum_{x \in O} \mathbbm{1}\{ \mathcal{A}(S^P, S^U) = 1\} \right. \right.\\
% & \quad \quad \quad \quad \quad \quad \quad \quad \left. \left.+ \frac{1 - \rho}{m - \claw} \sum_{x \in B \setminus \{O \cup S^P\}} \mathbbm{1}\{ \mathcal{A}(S^P, S^U) = 0\} \right] \right)
\end{aligned}
\end{equation}
We fix any $S^P \in O^{\nP}$. Furthermore, define event $E$ to be the event that $\left |S^U \mid O \right| \leq \frac{\claw}{2}$. The idea similar to Theorem~\ref{thm:real-scar-pos-lower-bound} is that, since $E$ has a significant probability, we can lower bound $\cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}}$ by conditioning it on event $E$ as
\begin{equation} \label{eq:unlabel-sam-lower-bound-3}
\begin{aligned}
& \cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)}} \\
& \geq
\cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \mid E}\Pr_{S^U \sim \D_{\X, O}^{\nU}}[E]}
\end{aligned}
\end{equation}
Next, we try to lower bound $\cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \mid E}}$. Fix any $O \in V$, notice that $\D_{\X, O}(x) = \D_{\X, O}(x')$ is the same for every $x, x' \in O$. Moreover, $\D_{\X, O}(x) = \D_{\X, O}(x')$ for every $x, x' \in B \setminus O$. Therefore, for every $S^U \in B^{\nU}$, and every $O, O' \in W(S^P)$ such that $S^U \mid O = S^U \mid O'$, we have
$$\Pr_{S \sim \D_{\X, O'}^{\nU}}[S = S^U \mid E] =\Pr_{S \sim \D_{\X, O}^{\nU}}[S = S^U \mid E].$$
Therefore, by fixing $S^U \in B^{\nU}$ and any $S'$ with $|S'| \leq \claw /2$ which represents $S^U \mid O$, we can define $P_{S^U, S'} :=\Pr_{S \sim \D_{\X, O}^{\nU}}[S = S^U \mid E]$.
This fact gives away the idea of grouping all the $O \in W(S^P)$ that have the same $S^U \mid O$ for a fix $S^U \in B^{\nU}$. Formally, we have
\begin{equation} \label{eq:unlabel-sam-lower-bound-4}
\begin{aligned}
& \cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \mid E}} \\
% & \cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\frac{\rho}{\claw} \sum_{x \in O} \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 1\} + \frac{1 - \rho}{m - \claw} \sum_{x \in B \setminus (O \cup S^P)} \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 0\}}}
& = \frac{1}{ |W(S^P)|} \sum_{O \in W(S^P)} \sum_{S^U \in B^{\nU}}\Pr_{S \sim \D_{\X, O}^{\nU}}[S = S^U \mid E]
{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} \\
& = \frac{1}{|W(S^P)| } \sum_{S^U \in B^{\nU}} \sum_{\substack{S' \in B^* \\ |S'| \leq \claw / 2}} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \Pr_{S \sim \D_{\X, O}^{\nU}}[S = S^U \mid E] \err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \\
& = \frac{1}{|W(S^P)| } \sum_{S^U \in B^{\nU}} \sum_{\substack{S' \in B^* \\ |S'| \leq \claw / 2}} P_{S^U, S'} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \\
\end{aligned}
\end{equation}
% Note that
% Thus,
% \begin{equation} \label{eq:unlabel-sam-lower-bound-5}
% \begin{aligned}
% & \frac{1}{\left(\begin{array}{c}m-\claw\\ \claw\end{array}\right) } \sum_{S^U \in B^{\nU}} \sum_{\substack{S' \in B^* \\ |S'| \leq \claw / 2}} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \Pr_{S \sim \D_{\X, O}^{\nU}}[S = S^U \mid E] \err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \\
% & = \frac{1}{\left(\begin{array}{c}m-\claw\\ \claw\end{array}\right) } \sum_{S^U \in B^{\nU}} \sum_{\substack{S' \in B^* \\ |S'| \leq \claw / 2}} P_{S^U, S'} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \\
% \end{aligned}
% \end{equation}
Next, fixing any $S^U \in B^U$ and any $S'$ with $|S'| \leq \claw / 2$. Note that since the error is always no less than the error restricted over $B \setminus (S^P \cup S^U)$, we derive
\begin{equation} \label{eq:unlabel-sam-lower-bound-6}
\begin{aligned}
& \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \\
& \geq \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \sum_{x \in B \setminus (S^P \cup S^U)} \left(\frac{\rho}{\claw}
\mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 1, x \in O \}
% & \quad \quad \quad \quad \quad \quad \quad \quad \left.
+ \frac{1 - \rho}{m - \claw} \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 0, x \notin O \} \right) \\
& = \sum_{x \in B \setminus (S^P \cup S^U)} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S'}} \left(\frac{\rho}{\claw}
\mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 1, x \in O \}
% & \quad \quad \quad \quad \quad \quad \quad \quad \left.
+ \frac{1 - \rho}{m - \claw} \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 0, x \notin O \} \right) \\
& = \sum_{x \in B \setminus (S^P \cup S^U)} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S'}} \left( \frac{\rho}{\claw} \frac{|O \setminus S'|}{|B \setminus (S^P \cup S^U)|} \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 1\}
\right.\\
& \quad \quad \quad \quad \quad \quad \quad \quad \left.
+ \frac{1 - \rho}{m - \claw} \frac{|B \setminus (S^P \cup S^U \cup O)|}{|B \setminus (S^P \cup S^U)|} \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 0\} \right)\\
\end{aligned}
\end{equation}
Where the last line is due to symmetry of $W(S^P)$. Next note that since $|S'| \leq \frac{\claw}{2}$ we have $\frac{|O \setminus S'|}{\claw} \geq \frac{1}{2}$. Moreover, since $m \geq 2(\nP + \nU) + \claw$, we have
$$
\frac{|B \setminus (S^P \cup S^U \cup O)|}{m - \claw} \geq \frac{m - \nP - \nU - \claw}{m - \claw} \geq \frac{1}{2}
$$
Combining these facts with \eqref{eq:unlabel-sam-lower-bound-6}, we derive
\begin{equation} \label{eq:unlabel-sam-lower-bound-7}
\begin{aligned}
& = \sum_{x \in B \setminus (S^P \cup S^U)} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S'}} \left( \frac{\rho \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 1\} + (1 - \rho) \mathbbm{1}\{ \mathcal{A}(S^P, S^U)(x) = 0\}}{2 |B \setminus (S^P \cup S^U)|} \right) \\
& \stackrel{(i)}{\geq} \sum_{x \in B \setminus (S^P \cup S^U)} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S'}} \frac{\rho}{2} \\
& = \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S'}} \frac{\rho}{2}
\end{aligned}
\end{equation}
Where (i) is due to the fact that $\rho \geq \frac{1}{2}$. Combining \eqref{eq:unlabel-sam-lower-bound-4}
% , \eqref{eq:unlabel-sam-lower-bound-5}
and \eqref{eq:unlabel-sam-lower-bound-7} we derive
\begin{equation} \label{eq:unlabel-sam-lower-bound-8}
\begin{aligned}
& \cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right) \mid E}} \\
& \geq \frac{1}{|W(S^P)| } \sum_{S^U \in B^{\nU}} \sum_{\substack{S' \in B^* \\ |S'| \leq \claw / 2}} P_{S^U, S'} \sum_{\substack{O \in W(S^P) \\ S^U \mid O = S' }} \frac{\rho}{2} \\
& = \frac{1}{ |W(S^P)| } \sum_{O \in W(S^P)} \sum_{S^U \in B^{\nU}}\Pr_{S \sim \D_{\X, O}^{\nU}}[S = S^U \mid E] \frac{\rho}{2}
\\
& = \cE{O \sim U_{W(S^P)}}{\cE{S^U \sim \D_{\X, O}^{\nU}}{\frac{\rho}{2} \mid E}} = \frac{\rho }{2}
\end{aligned}
\end{equation}
Next, we attempt to lower bound $\Pr_{S^U \sim \D_{\X, O}^{\nU}}[E]$. Similar to the proof of Theoreom~\ref{thm:real-scar-pos-lower-bound}, using Multiplicative Chernoff bound (Lemma~\ref{lem:mult-chern-bound}) for any $\gamma \in \left[0, \frac{\claw}{\rho} - 1\right]$ we derive
\begin{equation*}
\Pr_{S^U \sim \D_{\X, O}^{\nU}} \left[|S^U \mid O | \geq (1 + \gamma) \nU \rho \right] \leq e^{-\frac{\nU \rho \gamma^2}{3}}
\end{equation*}
We set $\gamma = 1$. Note that since $\varepsilon = \min \left[\frac{7}{2400}, \frac{7 \claw}{4800 \nU}\right]$, we get $\rho = \min \left[\frac{1}{2}, \frac{\claw}{4 \nU}\right]$. Thus, we get
$$
\Pr_{S^U \sim \D_{\X, O}^{\nU}} \left[|S^U \mid O | \geq \frac{\claw}{2} \right] \leq e^{-\frac{\claw}{12}} < 0.93.
$$
Therefore, $\Pr_{S^U \sim \D_{\X, O}^{\nU}}[E] > 0.07$. Combing this fact with \eqref{eq:unlabel-sam-lower-bound-1}, \eqref{eq:unlabel-sam-lower-bound-2}, \eqref{eq:unlabel-sam-lower-bound-3} and \eqref{eq:unlabel-sam-lower-bound-8} we derive
$$
\max_{O \in V} \cE{S^P \sim \D_{+, O}^{\nP}, S^U \sim \D_{\X, O}^{\nU}}{\err_{\D_O}\left(\mathcal{A}(S^P, S^U)\right)} > \frac{7}{600} \rho
$$
Thus, there exists a $O^\star \in V$ such that
$\cE{S^P \sim \D_{+, O^\star}^{\nP}, S^U \sim \D_{\X, O^\star}^{\nU}}{\err_{\D_{O^\star}}\left(\mathcal{A}(S)\right)} > \frac{7}{600} \rho$. Since the error is always less than $3 \rho$, and by using Lemma~\ref{lem:highprob-to-expect} we derive
$$
\begin{aligned}
\Pr_{S^P \sim \D_{+, O^\star}^{\nP}, S^U \sim \D_{\X, O^\star}^{\nU}}[\err_{\D_{O^\star}}\left(\mathcal{A}(S)\right) \geq \varepsilon] &\geq \frac{\frac{7}{600} \rho - \varepsilon }{ (3 \rho - \varepsilon)} \\
& = \frac{\varepsilon}{ \frac{3600}{7} \varepsilon - \varepsilon } \\
& = \frac{7}{3599} > \frac{1}{1000}.
\end{aligned}
$$
and this completes the proof.
\textit{Proof of $m^{unlabel}_\C(\varepsilon, \delta) = \Omega \left( \frac{\ln(1 / \delta)}{\varepsilon} \right)$.} Next we prove that for every $\nP \in \Nat$, $\varepsilon< \frac{1}{8}$, $\delta < \frac{1}{4}$, $\nU = \frac{\ln \left(\frac{1}{4 \delta}\right)}{2 \varepsilon}$ and PU learner $\mathcal{A}$ there is a distribution $\D$ realized by $\C$ over $\X \times \{0, 1\}$ such that
$$
\Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \varepsilon \right] > \delta.
$$
Let $m := 2(\nP + \nU + 1)$. Note that since claw number is more than $1$, there exists a $B \subseteq \X$ with size $m$ such that $B \setminus \{x\} \subseteq \C$ for all $x \in B$. For any $x \subseteq B$ and any $(x', y') \in \X \times \{0, 1\}$ we define the distribution $\D_x$ over $\X \times \{0, 1\}$ as
\begin{equation}
\D_x(\{(x', y')\}) := \begin{cases} \varepsilon & \text{if } x' = x \text{, and } y = 0 \\
\frac{(1 - \varepsilon)}{(m - 1)} & x' \in B \setminus \{x\} \text{, and } y = 1 \\
0 & \text{o.w. }
\end{cases}
\end{equation}
Note that all such $\D_x$ are realized by $\C$.
% Suppose that $S^P \sim \D_{+, x^\star}^{\nP}$ and $S^U \sim \D_{\X, x^\star}^\nU$ where $x^\star$ is drawn from $U_B$.
It is enough to show that
$$
\Pr_{x^\star \sim U_B, S^P \sim \D_{+, x^\star}^{\nP}, S^U \sim \D_{\X, x^\star}^{\nU}} \left[\err_{\D} \left( \mathcal{A}(S^P, S^U) \right) \geq \varepsilon \right] > \delta.
$$
Fix any $x^\star \in B$. Note that similar to the manner \eqref{eq:real-scar-pos-lower-bound-4} in the proof of Theorem~\ref{thm:real-sar-pos-lower-bound} was obtained, for all $\varepsilon < \frac{1}{2}$ we derive
\begin{equation} \label{eq:unlabel-sam-lower-bound-9}
\begin{aligned}
\Pr_{S^U \sim \D_{\X, x^\star}^{\nU}}[x^\star \notin S^U] &= (1 - \varepsilon)^{\nU} > e^{ - 2\varepsilon \nU} = 4 \delta.
\end{aligned}
\end{equation}
Next, consider any positive sample $S^P$ and unlabeled sample $S^U$ such that $S^U$ doesn't contain $x^\star$. Note that the probability of $S^P$ and $S^U$ is the same w.r.t. every $\D_x$ such that $x \in B \setminus (S^U \cup S^P)$. We consider two cases based on the number of negative labels $\mathcal{A}(S^P, S^U)$ predicts. We prove that for both cases the error of $\mathcal{A}(S^P, S^U)$ is more than $\varepsilon$ with probability at least $\frac{1}{4}$. Therefore, the probability that error of $\mathcal{A}(S^P, S^U)$ is at least $\epsilon$ would be more than
$$\frac{\Pr_{S^U \sim \D_{\X, x^\star}^{\nU}}[x^\star \notin S^U]}{4} = \delta,$$
which completes the proof.
Case 1: $\mathcal{A}(S^P, S^U)$ has more than $2 m \varepsilon + 1$ negative labels. Then, at least $2m \varepsilon $ of them are not $x^\star$, and subsequently, since $\varepsilon < \frac{1}{2}$ the error is guaranteed to be at least
$$
\frac{2 m \varepsilon (1 - \varepsilon)} {m - 1} > \varepsilon.
$$
Case 2: $\mathcal{A}(S^P, S^U)$ at most $2 m \varepsilon + 1$ negative labels. Note that, since $m = 2(\nP + \nU + 1)$, we have $|B \setminus (S^U \cup S^P)| \geq \frac{m}{2} + 1$. Therefore, since $\varepsilon < \frac{1}{8}$, $\mathcal{A}(S^P, S^U)$ predicts at least $m \left( \frac{1}{2} - 2 \varepsilon \right) \geq \frac{m}{4}$ positive labels over $B \setminus (S^U \cup S^P)$. Therefore, since $x^\star$ is drawn uniformly, with more than $\frac{1}{4}$ chance the label of $x^\star$ would be predicted positive, which indicates that $\mathcal{A}(S^P, S^U)$ has more than $\varepsilon$ error.
\end{proof}
\propcintclaw*
\begin{proof}
\textit{Only if side.} Since $\VCD(\C_\cap) = \infty$, for every $m \in \Nat$, there exists a subset $B = \{x_1, ..., x_m\}$ that is shattered by $\C_\cap$. Thus, for all $i \in [m]$ there should be a $c_{\cap} \in \C_{\cap}$ such that $c_{\cap} \cap B = B \backslash \{x_i\}$. This indicates that $B \backslash \{x_i\} \subseteq c_{\cap}$. Consequently, there should be a $c \in \C$ such that $c_{\cap} \subseteq c$, and $x_i \notin c$. Since $B \backslash \{x_i\} \subseteq c$, in turn, implies that $c \cap B = B \backslash \{x_i\}$. Thus, $\{O \subseteq B \mid |O| = m - 1\} \subseteq \C \mid B$, and therefore the claw number of $\C$ is at least 1.
\textit{If side:} Since claw number is at least 1, for every $m \in \Nat$ there exists a subset $B = \{x_1, ..., x_{m+1}\}$ such that for all $i \in [m + 1]$ we have $B \backslash \{x_i\} \in \C \mid B$. For every $i \in [m + 1]$ define $c_i$ to be the concept such that $c_i \cap B = B \backslash \{x_i\}$.
Denote $\bar B := B \setminus \{x_{m+1}\}$. Next, consider any $A \subseteq \bar B$ we prove that there exists a $c_{\cap} \in \C_{\cap}$ that satisfies $c_\cap \cap \bar B = A$, and this shows that $\bar B$ is shattered by $\C_{\cap}$. This implies that for every $m \in \Nat$ we have $\VCD(\C_{\cap}) \geq m$, which completes the proof. When $A = \bar B$ define $c_\cap:= c_{m + 1}$, and we derive $c_\cap \cap \bar B = \bar B$. Otherwise, define $c_\cap := \bigcap_{i: x_i \in \bar B \backslash A} c_i$. It is easy to see $c_\cap$ satisfies the desired property.
\end{proof}
\begin{lemma} \label{lem:liu2002-cor5}[Corollary 5 of \cite{liu2002partially}]
Let $\C$ be a concept class with VC-dimension d over $\X$. Let $S$ be an i.i.d. sample of size $n$ from a distribution $\D_{\X}$ over $\X$. There exists a constant $M > 1$, such that if
$n > M \left(\frac{d \ln (1 / \varepsilon) + \ln(1 / \delta)}{\varepsilon} \right)$, then for every $c, c^\star \in \C$ with probability $1 - \delta$ we have
$$
Pr\left(c(x) \neq c^\star(x)\right) > \frac{3 \sum_{x \in S} \mathbbm{1}\{ c(x) \neq c^\star(x) \}}{n} + \epsilon
$$
and
$$
\frac{\sum_{x \in S} \mathbbm{1}\{ c(x) \neq c^\star(x) \}}{n} >3 Pr \left(c(x) \neq c^\star(x)\right) +\epsilon
$$
\end{lemma}
\lemepnet*
\begin{proof}
Proof of this theorem was extracted from Theorem 1 of \cite{liu2002partially}. First, note that according to the definition of $\varepsilon-$net, for any $c$ such that $T \subseteq c$ we have that
\begin{equation} \label{eq:epnet-pi-1}
\begin{aligned}
\Pr(c(x) = 0, y = 1)
\leq \err^P_{\D}(c) \leq \varepsilon.
\end{aligned}
\end{equation}
% Let $S = \{(x, \clabel(x)) \mid x \in U\}$. Note that since $\err_{\D}(c^\star) = 0$ with probability 1, $S$ is consistent with $c^\star$.
Note that since $T \subseteq c^\star$, we have $|c^{PU} \mid S| \leq |c^{\star} \mid S|$. Thus,
$$
|c^{PU} \cap c^{\star} \mid S| + |c^{PU} \cap \overline{c^\star} \mid S| \leq |c^\star \mid S|
$$
Therefore,
\begin{equation} \label{eq:epnet-pi-2}
\begin{aligned}
|c^{PU} \cap \overline{c^\star} \mid S| & \leq |c^\star \mid S| - |c^{\star} \cap c^{PU} \mid S| \\
& = |\overline{c^{PU}} \cap c^\star \mid S|
\end{aligned}
\end{equation}
From Lemma~\ref{lem:liu2002-cor5} it can be deduced that there exists a $M > 1$ such that as long as $|S| \geq M \left(\frac{d \ln (1 / \varepsilon) + \ln(1 / \delta)}{\varepsilon}\right)$ with probability $1 - \delta$ we have
\begin{equation} \label{eq:epnet-pi-3}
\frac{|\overline{c^{PU}} \cap c^\star \mid S|}{|S|} \leq 3 Pr(c^\star(x) = 1, c^{PU}(x) = 0)) + \varepsilon
\end{equation}
Similarly as long as $|S| \geq M \left(\frac{d \ln (1 / \varepsilon) + \ln(1 / \delta)}{\varepsilon} \right)$ with probability $1 - \delta$ we have
\begin{equation} \label{eq:epnet-pi-4}
\Pr(c^\star(x) = 0, c^{PU}(x) = 1)) \leq 3\frac{|c^{PU} \cap \overline{c^\star} \mid S|}{|S|} + \varepsilon
\end{equation}
Combining \eqref{eq:epnet-pi-2}, \eqref{eq:epnet-pi-3} and \eqref{eq:epnet-pi-4} with probability $1 - 2\delta$ we have
\begin{equation} \label{eq:epnet-pi-5}
\begin{aligned}
\Pr(c^\star(x) = 0, c^{PU}(x) = 1)& \leq 9 \Pr(c^\star(x) = 0, c^{PU}(x) = 1) + 4 \varepsilon \\
& \stackrel{(i)}{\leq} 13 \varepsilon
\end{aligned}
\end{equation}
Where (i) is due to \eqref{eq:epnet-pi-1} and $\Pr( c^\star(x) \neq y ) = 0$. Thus combining \eqref{eq:epnet-pi-1} and \eqref{eq:epnet-pi-5} and the fact that $\Pr( c^\star(x) \neq y ) = 0$ with probability $1 - 2 \delta$ we derive
$$
\begin{aligned}
\err_{\D}(c^{PU}) &= \Pr(c^{PU}(x) \neq c^\star(x)) \\
& = \Pr(c^\star(x) = 0, c^{PU}(x) = 1) + \Pr(c^\star(x) = 0, c^{PU}(x) = 1) \\
&\leq 14 \varepsilon.
\end{aligned}
$$
Which completes the proof.
\end{proof}
% \corscarup*
\section{Missing Proofs from Section~\ref{sec:PU-real-beyond-scar}}
\thmsarup*
\begin{proof}
% Let $C = C_{\C \Delta \C}\left(\cP, \D_+ \right)$.
As mentioned in Corollary~\ref{cor:real-scar-PERM}, it is well-known that as long as $\nP > M \left(\frac{d \ln (1 / r \varepsilon) +\ln (1 / \delta)}{r \varepsilon} \right)$ for a constant $M$, $S^P$ is a $r \varepsilon-$net for $\C \Delta \C$ with respect to $\cP$ with probability $1 - \delta$. Using lemma~\ref{lem:ben2012-lem3} we derive that $S^P$ is a $\varepsilon-$net for $\C \Delta \C$ with respect to $\D_+$ with probability $1 - \delta$. Moreover, we know $\cP(A) = 0$ for every measurable $A$ that has $\D_{+}(A) = 0$. This indicates that given any $c^\star$ with $\err_{\D}(c^\star) = 0$, almost surely we have $S^P \subseteq c^\star$. Plugging these into lemma~\ref{lem:epnet-pi} completes the proof.
\end{proof}
\thmsarposlow*
\begin{proof}
Let $\rho = 320\varepsilon$, $B = \{x_1, ..., x_d\}$ be any set of size $d$ shattered by $\C$, and denote $\bar B := B \setminus \{x_d\}$. Consider the set of distributions $\mathcal{W}^{scar-pos}_{\rho, d} = \{(\D_O, \D_{+, O}) : O \subseteq \bar B\}$, as defined in the proof of Theorem~\ref{thm:real-scar-pos-lower-bound}. For each $O \subseteq \bar B$, define a distribution $\cP_O$ such that $\cP_O(\{(x, y)\}) := r \cdot \D_{+, O}(\{(x, y)\})$ for every $x \in \bar B$ and $y \in \{0, 1\}$, and assign the remaining probability mass of $\cP_O$ to the point $(x_d, 1)$. By construction, we have $R(\cP_O, \D_{+, O}) \geq r$.
% Moreover, let $\rho' := 320 \varepsilon r$. Then,
% \[
% \{\cP_O \mid O \subseteq B\} = \{\D_+ \mid \D \in \mathcal{G}^{\text{pos}}_{\rho', d} \}.
% \]
Now define the set of duos $\mathcal{W}^{sar}_{\rho, d} := \{(\cP_O, \D_O) \mid O \subseteq \bar B\}$. By denoting $\rho' := 320 \varepsilon r$, it is easy to see that PU learnability over $\mathcal{W}^{sar}_{\rho, d}$ is equivalent with PU learnability over $\mathcal{W}^{scar-pos}_{\rho', d}$. Consequently, from the argument in the proof of Theorem~\ref{thm:real-scar-pos-lower-bound}, for $d \geq 9$ the number of positive examples required by any algorithm that PU learns $\C$ over $\mathcal{W}^{sar}_{\rho, d}$ must satisfy
\[
m^{\text{pos}}_{\C}(\varepsilon, \tfrac{1}{319}) \geq \frac{d - 1}{512 \varepsilon r}.
\]
For $d \geq 2$ an identical argument also gives
\[
m^{\text{pos}}_{\C}(\varepsilon, \delta) \geq \frac{\ln(1 / 2\delta)}{2 r \varepsilon}.
\]
This completes the proof.
\end{proof}
% \subsection{Proof of Theorem~\ref{thm:real-dist-shift-lower-bound}}
\thmcovlow*
We obtain our lower bound by reducing the following problem to the PU learning problem:
\textbf{The Left/Right Problem.} We consider the problem of distinguishing two distributions from finite samples. The Left/Right Problem was introduced in \cite{kelly2010universal}: \\
\textbf{Input:} Three finite samples, $L, R$ and $M$ of points from some domain set $\mathcal{X}$. \\
\textbf{Output:} Assuming that $L$ is an i.i.d. sample from some distribution $P$ over $\mathcal{X}$, that $R$ is an i.i.d. sample from some distribution $Q$ over $\mathcal{X}$, and that $M$ is an i.i.d. sample generated by one of these two probability distributions, was $M$ generated by $P$ or by $Q$ ?
\textit{Lower bound for the Left/Right problem}: We say that a (randomized) algorithm
$(\delta, l, r, m)$-solves the Left/Right problem if, given samples $L$, $R$ and $M$ of sizes $l$, $r$ and $m$ respectively, it gives the correct answer with probability at least $1-\delta$. Lemma 1 of \cite{ben2012hardness} shows that for any sample sizes $l$, $r$ and $m$ and for any $\gamma < 1/2$, there exists a finite domain $\mathcal{X} = \{1, 2, \dots, n\}$ and a finite class $\mathcal{W}_n^{uni}$ of triples of distributions over $\mathcal{X}$ such that no algorithm can $(\gamma, l, r, m)$-solve the Left/Right problem for this class. In this class, both the distribution generating $L$ and the distribution generating $R$ are uniform over half of the points in $\X$, but their supports are disjoint. More formally,
\[
\begin{aligned}
& \mathcal{W}_n^{uni} = \{(U_A, U_B, U_C) : A \cup B = \{1, \dots, n\}, A \cap B = \emptyset, |A| = |B|, \text{ and } C = A \text{ or } C = B\},
\end{aligned}
\]
where, for a finite set $Y$, $U_Y$ denotes the uniform distribution over $Y$.
\begin{lemma}[Lemma 1 of \cite{ben2012hardness}] \label{lem:LR-lowerbound-ben2012}
For any given sample sizes $l$ for $L$, $r$ for $R$ and $m$ for $M$ and any $0 < \gamma < 1/2$, if $k = \max\{l, r\} + m$, then for $n > (k+1)^2/(1-2\gamma)$ no algorithm has a probability of success greater than $1 - \gamma$ over the class $\mathcal{W}_n^{uni}$
\end{lemma}
\textit{Reducing the Left/Right problem to PU learning:} In order to reduce the Left/Right problem to PU learning, we define a class of PU learning problems that corresponds to the class of duos $\mathcal{W}_n^{uni}$, for which we have proven a lower bound on the sample sizes needed for solving the Left/Right problem. Let $\X$ be some domain of size $n$. Let $Y$ be the first $n / 3$ elements of $\X$ and $Z$ to be the next $2n / 3$ elements of it. Let $\C_{0, 1} = \{c_0, c_1\}$ where $c_1 = \X$ and $c_0 = Z$. Clearly $\VCD(\C_{0, 1}) = 1$. We define $\mathcal{W}_n^{cov}$ to be the class of duos $(\cP, \D)$, where $\D$ is a distribution with a deterministic label such that $\D_{\X}$ is uniform either over $Y \cup J$ or $Y \cup (Z \backslash J)$ for some uniform subset $J$ of $Z$ of size $n/3$, and $\clabel$ assigns points in $Y \cup J$ to 1 and points in $Z \backslash J$ to 0, and $\cP$ is uniform over $Y \cup J$. Notice that $\cP \in \mathcal{K}^{cov}_\D$. Note that we always either have $R(\cP, \D_+) = 1/2$ or $R(\cP, \D_+) = 1$. Moreover, since the label of $Y$ is always 1, $\D[\{(x,1): x \in \X\}] \geq 1/2$.
% and $d_{\C_{1, 0} \Delta \C_{\{1, 0\}}}(\cP, \D_+) = 0$
Furthermore, for all $(\cP, \D)$ in $\mathcal{W}_n^{cov}$
\[
\min_{c \in \C_{0, 1}} \err_\D(c) = 0
\]
for all elements of $\mathcal{W}_n^{cov}$.
\begin{lemma} \label{lem:LR-reduc-PU}
Consider any $s, t \in \Nat$. The Left/Right problem reduces to Domain Adaptation. More precisely, given a number $n$, suppose there exists a PU learner $\mathcal A$ such that for all $(\cP, \D) \in \mathcal{W}_{3n/2}^{cov}$, $\nP \geq s$ and $\nU \geq t$ satisfies
$$
\cE{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}}{\err_\D(\mathcal{A}(S^P, S^U) \geq \varepsilon } \leq \delta.
$$
Then, we can construct an algorithm that $(2\varepsilon + \delta, s, s, t + 1)$-solves the Left/Right problem on $\mathcal{W}_n^{uni}$.
\end{lemma}
\begin{proof}
Assume we are given samples $L=\left\{l_1, l_2, \ldots, l_s\right\}$ and $R=\left\{r_1, r_2, \ldots, r_s\right\}$ of size $s$ and a sample $M$ of size $t+1$ for the Left/Right problem coming from a triple $\left(U_A, U_B, U_C\right)$ of distributions in $\mathcal{W}_n^{uni}$. Then consider any set $Y$ of size $n / 2$ distinct from $A$ and $B$. We create the set $I^U$ in the following manner. Initiate $I^U = \varnothing$. Keep flipping an unbiased coin until $t$ of them are head, and each time the outcome of the coin was tail sample a $z \sim U_Y$ and add it to $I^U$. Similarly, initiate $I^P = \varnothing$, and keep flipping an unbiased coin until $s$ of them are head, and each time the outcome of the coin was tail sample a $z \sim U_{Y}$ and add it to $I^P$.
We construct an input to PU learning problem by setting the unlabeled sample $S^U=M \backslash\{p\} \cup I^U$, where $p$ is a point from $M$ chosen uniformly at random, and setting the positive sample $S^P = R \cup I^P$. Observe that $|S^P| \geq s$ and $|S^U| \geq t$.
% Note that $|I^P|$ is a negative binomial distribution with parameter $s$ and $\frac{1}{2}$. Thus, expectation and variance of $|I^P|$ are respectively $s$ and $2s$.
% Using Chebishev's inequality (see Theorem~\ref{thm:chebyshev}), we derive $Pr(|I^P| \geq 2s) \leq \frac{2s}{s^2} \leq \frac{1}{8}$. Similarly, we derive $Pr(|I^U| \geq 2t) \leq \frac{1}{8}$
These sets can now be considered as an input to the PU learning problem generated from a sampling positive distribution $\cP = U_{Y \cup B}$, and distribution $\D$ such that $\D_{\X}$ equal to $U_{Y \cup A} $ or to $U_{Y \cup B}$ (depending on whether $M$ was a sample from $U_A$ or from $U_B$ ) with labeling function being $\clabel(x)=0$ if $x \in A$ and $\clabel(x)=1$ if $x \in Y \cup B$. Observe that we have $R(\cP, \D_+) = 1/2$ or $R(\cP, \D_+) = 1$, and $\min_{c \in \C_{0, 1}} \err_\D(c) = 0$, and $\left(\cP, \D\right) \in \mathcal{W}_{3n/2}^{cov}$. Denote $c = \mathcal{A}(S^P, S^U)$. The algorithm for the Left/Right problem then outputs $U_A$ if $c(p)=0$ and $U_B$ if $c(p)=1$. Note that $\err_{\D} \left( c \right) \leq \varepsilon$ with confidence $1-\delta$. Thus, $\err_{U_M} \leq 2 \varepsilon$ with confidence $1 - \delta$. Therefore, the algorithm is correct with probability at least $2 \varepsilon + \delta$.
% Since with probability $1/4$ we have $S^U \leq 3 t$ and $S^P \leq 3s$, the proof is completed.
\end{proof}
\textit{Proof of Theorem~\ref{thm:real-dist-shift-lower-bound}.} Combining Lemma~\ref{lem:LR-reduc-PU} and Lemma~\ref{lem:LR-lowerbound-ben2012} we conclude that there exists no $\mathcal{A}$ such that for all $(\cP, \D) \in \mathcal{W}_{3n/2}^{cov}$ satisfies
$$
\cE{S^P \sim \cP^{\nP}, S^U \sim \D_\X^{\nU}}{\err_\D(\mathcal{A}(S^P, S^U) \geq \varepsilon } \leq \delta,
$$
unless $\nP + \nU \geq \sqrt{(1-2(2\varepsilon+\delta))n} - 2$, which completes the proof.
% show that no algorithm can solve the PU learning problem for $\mathcal{W}_{3n/2}^{cov}$, even under the assumption of realizability by $\C_{1,0}$, if . This completes the proof.
\thmcovlowlip*
\begin{proof}
Let $\mathcal{J} \subseteq \mathcal{X}$ be the points of a grid in $[0,1]^k$ with distance $1/\lambda$. Then we have $|\mathcal{J}| = (\lambda + 1)^k$. Then the class $\mathcal{W}_{\lambda}$ contains all duos $(\cP, \D)$, where the support of $\cP$ and $\D$ is $\mathcal{J}$,
$\D$ has deterministic labels and is realized by $\C_{0, 1}$ and $\cP \in \mathcal{K}^{cov}_{\D}$,
$C(\cP, \D_+) = 1/ 2$, and $\D[\{(x,1): x \in \X\}] \geq 1/2$,
% $d_{\C_{0, 1}, \C_{0, 1}}(\cP, \D_+) = 0$,
% and $\min_{c \in \C_{0, 1}} \err_{\D}(c) = 0$
with any arbitrary labeling functions $l : \mathcal{J} \to \{0,1\}$, as every such function is $\lambda$-Lipschitz. As $\mathcal{J}$ is finite, the bound follows from Theorem~\ref{thm:real-dist-shift-lower-bound}.
\end{proof}
% \subsection{Proof of Theorem~\ref{thm:real-dist-shift-PU}}
\thmcovup*
\begin{proof}
Let $\varepsilon>0$ and $\delta>0$ be given. We set $\varepsilon^{\prime}=\varepsilon / 28$ and $\delta^{\prime}=\delta / 6$ and divide the space $\X$ up into heavy and light boxes from $\mathcal{B}$, by defining a box $A \in \mathcal{B}$ to be light if $\D_+(A) \leq \varepsilon^{\prime} /|\mathcal{B}|=\varepsilon^{\prime} /(\sqrt{k} / \gamma)^k$ and heavy otherwise. We let $\mathcal{X}^l$ denote the union of the light boxes and $\mathcal{X}^h$ the union of the heavy boxes. Further, we let $\cP_h$ and $\D_{+, h}$ denote the restrictions of the $\cP$ and $\D_+$ to $\mathcal{X}^h$, i.e. we have $\cP_h(A)=\cP(A) / \cP\left(\mathcal{X}^h\right)$ and $\D_{+, h}(A)=$ $\D_+(A) / \D_+\left(\mathcal{X}^h\right)$ for all $A \subseteq \mathcal{X}^h$ and $\cP_h(A)=\D_{+, h}(A)=0$ for all $A \nsubseteq \mathcal{X}^h$. As $|\mathcal{B}|=(\sqrt{k} / \gamma)^k$, we have $\D_+\left(\mathcal{X}^h\right) \geq 1-\varepsilon^{\prime}$ and thus, $\cP\left(\mathcal{X}^h\right) \geq r \left(1-\varepsilon^{\prime}\right)$. We will show that \\
\textbf{Claim 1.} With probability $1 - \delta'$ we have $S^U$ hits every heavy box (Similar to claim 1 of Theorem 3 of \cite{ben2012hardness}).
\textbf{Claim 2.}
With probability at least $1-2 \delta^{\prime}$ the intersection of $S^P$ and $\mathcal{X}^h$ is an $\varepsilon^{\prime}$-net for $\C \Delta \C$ with respect to $\D_{+, h}$ (claim 2 of Theorem 3 of \cite{ben2012hardness}). \\
To see that these imply the claim of the theorem, let $S^h=S^P \mid \mathcal{X}^h$ denote the intersection of the source sample and the union of heavy boxes. By Claim 1, $S^U$ hits every heavy box with high probability, thus $S^h \subseteq S^{\prime}$, where $S^{\prime}$ is the intersection of $S^P$ with boxes that are hit by $S^U$ (see the description of the algorithm $\mathcal{A})$. Therefore, since $S^h$ is an $\varepsilon^{\prime}$-net for $\C \Delta \C$ with respect to $\D_{+, h}$, then so is $S^{\prime}$. Hence, with probability at least $1-3 \delta^{\prime}=1-\delta/2$ the set $S^{\prime}$ is an $\varepsilon^{\prime}$-net for $\C \Delta \C$ with respect to $\D_{+, h}$. Now note that an $\varepsilon^{\prime}$-net for $\C \Delta \C$ with respect to $\D_{+, h}$ is an $2\varepsilon^\prime$-net with respect to $\D_+$ as every set of $\D_+$-weight at least $2\varepsilon^\prime$ has $\D_{+, h}$ weight at least $\varepsilon^{\prime}$, by definition of $\mathcal{X}^h$ and $\D_{+, h}$.
% Finally, we need to show that $S^{\prime}$ being an $\varepsilon$-net for the set $\C \Delta \C$ of symmetric differences with respect to the target class, suffices for the $PERM(S^U, S^{\prime})$. to have target error at most $\varepsilon$.
Next, let $c^\star \in \C$ denote the $\gamma$-margin classifier with $\err_{\D}(c^\star) = 0$. We show that $S^P \subseteq c^\star$. Note that every box in $\mathcal{B}$ is labeled homogeneously with label 1 or label 0 by the labeling function $\clabel$ as $\clabel$ is a $\gamma$-margin classifier as well. Let $s \in S^{\prime}$ be a sample point and $A_s \in \mathcal{B}$ be the box that contains $s$. As $c^\star$ is a $\gamma$-margin classifier and $\D_{\X}\left(A_s\right)>0\left(A_s\right.$ was hit by $S^U$ by the definition of $\left.S^{\prime}\right)$, $A_s$ is labeled homogeneously by $c^\star$ as well and as $\err_{\D}(c^\star) = 0$ this label has to correspond to the labeling by $\clabel$. Thus $c^\star(s)=\clabel(s) = 1$ for all $s \in S^{\prime}$. This means that $S' \subseteq c^\star$. According to Lemma~\ref{lem:epnet-pi} for some $M > 1$ since $S' \subseteq c^\star$ and $\nU \geq M \left(\frac{d \ln( 1 / \varepsilon)+\ln (1 / \delta)}{\varepsilon}\right)$ as long as $S'$ is $\varepsilon/14$-net for $\C \Delta \C$ w.r.t. $\D_+$, with probability $1 - \delta/2$ we have $\err_\D(c) \leq \varepsilon$. This completes the proof.
\textbf{Proof of Claim 1:} Let $A$ be a heavy box, thus $\D_+(A) \geq \varepsilon^{\prime} /|\mathcal{B}|$, therefore $\D_{\X}(A) \geq \pi \cdot \varepsilon^{\prime} / |\mathcal{B}|$. Then, when drawing an i.i.d. sample $S^U$ from $\D_{\X}$, the probability of not hitting $A$ is at most $\left(1-\left(\pi\varepsilon^{\prime} /|\mathcal{B}|\right)\right)^{\nU}$. Now the union bound implies that the probability that there is a box in $\mathcal{B}^h$ that does not get hit by $X_U$ is bounded by
$$
\begin{aligned}
\left|\mathcal{B}^h\right|\left(1-\left(\pi \cdot \varepsilon^{\prime} /|\mathcal{B}|\right)\right)^{\nU} & \leq|\mathcal{B}|\left(1-\left(\pi \cdot \varepsilon^{\prime} /|\mathcal{B}|\right)\right)^{\nU}\\
& \leq|\mathcal{B}| \mathrm{e}^{-\pi \cdot\varepsilon^{\prime} \cdot \nU /|\mathcal{B}|}
\end{aligned}
$$
Thus, if $\nU \geq \frac{|\mathcal{B}| \ln \left(|\mathcal{B}| / \delta^{\prime}\right)}{\pi \cdot \varepsilon^{\prime}}=\frac{28(\sqrt{k} / \gamma)^k \ln \left(6(\sqrt{k} / \gamma)^k / \delta\right)}{\pi \varepsilon}$, then $S^U$ will hit every heavy box with probability at least $1-\delta^{\prime}$.
\textbf{Proof of Claim 2:} Let $S^h:=S^P \mid \mathcal{X}^h$. Note that, as $S^P$ is an i.i.d. $\cP$ sample, we can consider $S^h$ to be an i.i.d. $\cP_h$ sample. We have the following bound on the weight ratio between $\cP_h$ and $\D_{+, h}$ :
$$
\begin{aligned}
R_{\mathcal{I}}\left(\cP_h, \D_{+, h}\right)
& =\inf _{A \in \mathcal{I}, \D_{+, h}(A)>0} \frac{\cP_h(A)}{\D_{+, h}(A)}\\
& =\inf _{A \in \mathcal{I}, \D_{+, h}(A)>0} \frac{\cP(A)}{\D_+(A)} \frac{\D_+\left(\mathcal{X}^h\right)}{\cP\left(\mathcal{X}^h\right)} \\
& \geq r \frac{\D_+ \left(\mathcal{X}^h\right)}{\cP \left(\mathcal{X}^h\right)} \geq r\left(1-\varepsilon^{\prime}\right)
\end{aligned}
$$
where the last inequality holds as $\D_+\left(\mathcal{X}^h\right) \geq\left(1-\varepsilon^{\prime}\right)$ and $\cP\left(\mathcal{X}^h\right) \leq 1$. Note that every element in $\C \Delta \C$ can be partitioned into elements from $\mathcal{I}$, therefore we obtain the same bound on the weight ratio for the symmetric differences of $\C$ : $R_{\C \Delta \C}\left(\D_{+, h}, \D_{+, h}\right) \geq r\left(1-\varepsilon^{\prime}\right)$.
As we argued in Corollary~\ref{cor:real-scar-PERM}, it is well known that there is a constant $M>1$ such that, conditioned on $S^h$ having size at least $n':=M\left(\frac{d \ln \left(1 / \left(r (1-\varepsilon^{\prime}) \varepsilon^{\prime}\right)\right) +\ln \left(1 / \delta^{\prime}\right)}{r\left(1-\varepsilon^{\prime}) \varepsilon^{\prime}\right.} \right)$, with probability at least $1-\delta^{\prime}$ it is a $r\left(1-\varepsilon^{\prime}\right) \varepsilon^{\prime}$-net with respect to $\cP_h$ and thus an $\varepsilon^{\prime}$-net with respect to $\D_{+, h}$ by Lemma~\ref{lem:ben2012-lem3}.
Thus, it remains to show that with probability at least $1-\delta^{\prime}$ we have $\left|S^h\right| \geq n'$. As we have $\cP\left(\mathcal{X}^h\right) \geq r\left(1-\varepsilon^{\prime}\right)$, we can view the sampling of the points of $S^P$ and checking whether they hit $\mathcal{X}^h$ as a Bernoulli variable with mean $\mu=$ $\cP\left(\mathcal{X}^h\right) \geq r\left(1-\varepsilon^{\prime}\right)$. Thus, by Hoeffding's inequality (see Theorem~\ref{thm:Hoeffding}) we have that for all $t>0$, $\Pr\left(\mu|S^P|-\left|S^h\right| \geq t|S^P|\right) \leq \mathrm{e}^{-2 t^2|S^P|}$. If we set $r^{\prime}=r\left(1-\varepsilon^{\prime}\right)$, assume $|S^P| \geq \frac{2 n'}{r^{\prime}}$ and set $t=r^{\prime} / 2$, we obtain $\Pr\left(\left|S^h\right|< n'\right) \leq \Pr\left(\mu|S^P|-\left|S^h\right| \geq \frac{r^{\prime}}{2}|S^P|\right) \leq \mathrm{e}^{-\frac{r^{\prime 2}|S^P|}{2}}$.
Now $|S^P| \geq \frac{2 n'}{r^{\prime}}>\frac{2\left(d \ln \left(1 / \left(r (1-\varepsilon^{\prime}) \varepsilon^{\prime}\right)\right)+\ln \left(1 / \delta^{\prime}\right)\right)}{r^2\left(1-\varepsilon^{\prime}\right)^2 \varepsilon^{\prime}}$ implies that $\mathrm{e}^{-\frac{r^{\prime 2}|S^P|}{2}} \leq \delta^{\prime}$, thus we have shown that $S^h$ is an $\varepsilon^{\prime}$-net of $\C \Delta \C$ with probability at least $\left(1-\delta^{\prime}\right)^2 \geq 1-2 \delta^{\prime}$.
\end{proof}
\section{Missing Proofs from Section~\ref{sec:PU-agnostic}}
% \subsection{Proof of Theorem~\ref{thm:agnostic-PU-lower-bound}}
\thmagnolow*
We prove the theorem by reducing it to a problem we refer to as the \emph{generalized weighted die problem}. This approach is inspired by Theorem 1 of \cite{ben2011learning}, in which a learning problem --referred to as learning a classifier when a labeling is known (KLCL)-- is reduced to the \emph{weighted die problem}.
In the weighted die problem, a die has one face that is slightly biased, and the goal is to identify the biased face using $m$ rolls. In the generalized weighted die problem, multiple faces of the die may be slightly biased, and the goal is to identify all of them.
We now formally define the generalized weighted die problem. Before doing so, we introduce some necessary notation. For any $k \in \mathbb{N}$, define the set $V_k := 2^{[k]} \setminus \{[k], \varnothing\}$. Next, define two weighting functions $w^{-}, w^{+}: V_k \rightarrow [0, 1]$ as follows: for any $O \in V_k$ with $|O| \leq \frac{k}{2}$, set
\[
w^{-}(O) := 1 \quad \text{and} \quad w^+(O) := \frac{|O|}{k - |O|},
\]
and for $|O| > \frac{k}{2}$, set
\[
w^{-}(O) := \frac{k - |O|}{|O|} \quad \text{and} \quad w^+(O) := 1.
\]
\begin{definition} [Generalized Weighted Die Problem]
We define the generalized weighted die problem with parameters $k \geq 2$ and $\varepsilon \in (0, 1)$ as follows. For each $O \in V_k$ suppose there is a die with $k$ faces, with probability of each face $j \in [k]$ being
$$
P_O(j)= \begin{cases}
\frac{1 - w^{-}(O) \varepsilon}{k} & j \in O \\
\frac{1 + w^{+}(O) \varepsilon}{k} & j \notin O
\end{cases}
$$
The die $O$ is rolled $m$ times. A learner gets the outcome of these $m$ die rolls, and its target is to output a $h \in \{0, 1\}^{k}$ which minimizes $err(h) := \frac{1}{k} \left( \sum_{j \in O} h_j w^{-}(O) + \sum_{j \notin O} (1 - h_j) w^{+}(O) \right)$.
\end{definition}
% The idea behind lower bounding the sample complexity of agnostic PU learning is to reduce the generalized weighted die problem to
% the given learning task and apply a sample size lower bound for the generalized weighted die problem.
\begin{theorem} \label{thm:gen-weighted-die-sam}
Suppose the die $O$ in the generalized weighted die problem with parameters $k \geq 32$ and $\varepsilon \in (0, 1)$ is drawn uniformly at random from the set $V_k$. If the number of rolls is at most $k \left\lfloor \frac{1}{2 \varepsilon^2} \right\rfloor$, then any algorithm for the generalized weighted die problem incurs an error of at least $\frac{1}{160}$ with probability greater than $\frac{1}{320}$.
\end{theorem}
\begin{proof}
Set $m = k \lfloor \frac{1}{2 \varepsilon^2} \rfloor$. Note that it is multiplied by $k$. With an argument similar to Lemma 5.1 of \cite{anthony2009neural} and Theorem 2 of \cite{ben2011learning}, it is easy to see that the perfect learner $L_0$ predicts $j$ to have positive bias iff the number of samples from $j$ is bigger than $m / k$.
For a sample roll drawn from $P_O^m$ define $c_S$ to be the output of $L_0$. Define event $E$ to be the event that $\min(|O|, k - |O|) \geq \frac{k}{4}$
\begin{equation} \label{eq:gen-weighted-die-sam-1}
\begin{aligned}
& \cE{O \sim U_{V_k}}{\cE{S \sim P_O^m }{err(c_S)}} \\
& = \cE{O \sim U_{V_k}}{\cE{S \sim P_O^m }{err(c_S)} \mid E} \Pr_{O \sim U_{V_k}} \left [E \right] \\
\end{aligned}
\end{equation}
Note that since two set in $2^{[k]} \setminus V_k$ are not in event $E$ it is clear that $\Pr_{O \sim U_{V_k}} \left [E \right] \geq Pr_{O \sim U_{2^{[k]}}} [E]$. Moreover, note that when $O$ is chosen uniformly from $U_{2^{[k]}}$, $|O|$ can be looked upon as a random variable drawn from $Bin(k, 1/2)$. Therefore, as long as $k \geq 32$, using Multiplicative Chernoff bound (Lemma~\ref{lem:mult-chern-bound}) we can derive
\begin{equation} \label{eq:gen-weighted-die-sam-2}
\Pr_{O \sim U_{2^{[k]}}} \left [ E \right] \geq \left(1 - 2 \exp\left( - \frac{k}{16}\right)\right) > \frac{1}{2}
\end{equation}
Then we try to bound $\cE{O \sim U_{V_k}}{\cE{S \sim P_O^m }{err(c_S)} \mid E }$. For every $i \in [k]$ denote $s_i$ to be the number of $i$ in $S$. Then fix any $O \in V_k$ such that $\min(|O|, k - |O|) \geq \frac{k}{4}$, and any $i \in O$. Note that $L_0$ will make a wrong prediction for the face $i$ iff $s_i \geq m / k$. Since $s_i$ is $Bin(p, m)$ where $p = \frac{1 - w^{-}(O) \varepsilon}{k}$, using Slud's inequality (Lemma~\ref{lem:slud}) we can derive
\begin{equation} \label{eq:gen-weighted-die-sam-3}
\begin{aligned}
\Pr[s_i \geq m / k] & \geq P\left[ Z \geq \frac{m \varepsilon w^{-}(O)}{ \sqrt{m (1 + \varepsilon) (k - 1 + \varepsilon)}} \right] \\
& \stackrel{(i)}{\geq} P\left[ Z \geq \frac{m \varepsilon}{ \sqrt{m (k - 1)}} \right] \\
& \geq P\left[ Z \geq \sqrt{\frac{ 2m \varepsilon^2 }{ k}} \right] \\
& \stackrel{(ii)}{\geq} \frac{1}{2} \left(1 - \sqrt{1 - \exp \left(-\frac{ 2m \varepsilon^2 }{ k}\right) } \right) \\
& \geq \frac{1}{2} \left(1 - \sqrt{1 - e^{-1}} \right) > 0.1
\end{aligned}
\end{equation}
Where $Z \sim N(0,1)$ is a normally distributed random variable with mean of 0 and standard deviation of 1. Moreover, (i) is due to $w^-(O) \leq 1$ and $\varepsilon \geq 0$, and (ii) is due to Lemma~\ref{lem:norm-tail-bound}. Thus,
\begin{equation} \label{eq:gen-weighted-die-sam-4}
\begin{aligned}
\cE{S \sim P_O^m}{err(c_S)} & \geq \sum_{i \in O} \frac{\Pr[s_i \geq m / k]}{k} \\
& > |O| \cdot w^-(O) \cdot \frac{0.1}{k} \\
& = \min(|O|, k - |O|) \cdot \frac{0.1}{k} \\
& \geq \frac{1}{40}
\end{aligned}
\end{equation}
Combining this with \eqref{eq:gen-weighted-die-sam-1}, \eqref{eq:gen-weighted-die-sam-2}, we conclude that $\mathbbm{E}_{O \sim U_{V_k}}[\cE{S \sim P_O^m }{err(c_S)}] > \frac{1}{80}$. Using Lemma~\ref{lem:highprob-to-expect} since $err(c_S)$ is always less than 2, we have
$$Pr_{O \sim U_{V_k}, S \sim P^O_m} \left [err(c_S) \geq \frac{1}{160} \right] > \frac{1}{320}$$
which completes the proof.
\end{proof}
Consider the specific case where $k = 2$. Then $V_2 = \{\{1\}, \{2\}\}$. In this case, one of the faces always has probability $p = \frac{1 + \varepsilon}{2}$, and the other has probability $p = \frac{1 - \varepsilon}{2}$. Any algorithm with error less than $\frac{1}{2}$ must correctly identify which face has the higher probability. In other words, in this special case, the generalized weighted die problem reduces to Lemma 5.1 of \cite{anthony2009neural}, stated below.
\begin{lemma} [Lemma 5.1 of \cite{anthony2009neural}] \label{lem:ber-sample}
Let $\varepsilon < \frac{1}{2}$, $\delta< \frac{1}{4}$. Suppose $y = U_{\{-1, +1\}}$. Then if $m < \frac{1}{4 \varepsilon^2} \ln\left( \frac{1}{4 \delta}\right)$ there is no algorithm that can predict $y$ with probability more than $1 - \delta$ using a sample $S \sim Bin(m, \prior)$ where $\prior = \frac{1 + y \varepsilon}{2}$.
\end{lemma}
\begin{corollary} \label{cor:gen-weighted-die-2}
Let $\varepsilon < \frac{1}{2}$, $\delta< \frac{1}{4}$. Suppose the die $O$ in the generalized weighted die problem with parameters $k = 2$ and $\varepsilon$ is drawn uniformly at random from the set $V_2$. If the number of rolls is at most $\frac{1}{4 \varepsilon^2} \ln\left( \frac{1}{4 \delta}\right)$, then any algorithm for the generalized weighted die problem incurs an error of at least $\frac{1}{2}$ with probability greater than $\delta$.
\end{corollary}
\textit{Proof of Theorem~\ref{thm:agnostic-PU-lower-bound}.}
The upper bounds for Theorem~\ref{thm:agnostic-PU-lower-bound} are already known \cite{du2015convex}. We only focus on the lower-bounds.
Consider any $k \in \Nat$, and consider any $B \subseteq \X$ of size $2k$, and any $\rho \in (0,1)$. We randomly divide $B$ into two halves of size $k$ named $B^1 = \{x_1^1, x_2^1, ..., x_k^1\}$ and $B_2 = \{x_1^1, x_2^2, ..., x_k^2\}$. Let $\rho \in (0, 1)$, and for any $O^1, O^2 \in V_k$, we define distribution $\D_{O^1, O^2}$ over $\X \times \{0, 1\}$ for all $(x, y) \in \X \times \{0, 1\}$ as
$$
\D_{O^1, O^2}(x, y) = \begin{cases}
\frac{1}{4 k} & \text{if } x \in B_1, \text{ and } y = 1 \\
\frac{1 + w^+(O^1) \rho}{4 k} & \text{if } x = x^1_i \text{ and } i \notin O^1 \text{ and } y = 0 \\
\frac{1 - w^-(O^1) \rho}{4 k} & \text{if } x = x^1_i \text{ and } i \in O^1 \text{ and } y = 0 \\
\frac{1 + \frac{(2y - 1) w^+(O^2) \rho}{2} }{4 k} & \text{if } x = x^2_i \text{ and } i \notin O^2 \\
\frac{1 - \frac{(2y - 1) w^-(O^2) \rho}{2} }{4 k} & \text{if } x = x^2_i \text{ and } i \in O^2 \\
0 & \text{o.w.}
\end{cases}
$$
We define $\mathcal{W}^{agno}_{\rho, B} := \{ (\D_{O^1, O^2}, \D_{+, O^1, O^2}) \mid O^1, O^2 \in V_k\}$. Notice that every $\D_{O^1, O^2}$ satisfies $\D_{O^1, O^2}[\{(x,1): x \in \X\}] = \frac{1}{2}$. Moreover, it is easy to see that the error with respect to $\D_{O^1, O^2}$ is minimized by any function $f:\X \rightarrow \{0, 1\}$ that satisfies $f \cap B^1 = O^1$ and $f \cap B^2 = B^2 \setminus O^2$. Therefore, for any function $c$ we have
\begin{equation} \label{eq:excess-risk}
\begin{aligned}
\err_{\D_{O^1, O^2}}(c) - \min_{\text{functions }f} \err_{\D_{O^1, O^2}}(f) & = \frac{\rho}{4k} \left( \sum_{i \in O^1} \mathbbm{1}[c(x^1_i) \neq 1] w^-(O^1) + \sum_{i \notin O^1} \mathbbm{1}[c(x^1_i) \neq 0] w^+(O^1) \right. \\
& \left. \quad
+ \sum_{i \in O^2} \mathbbm{1}[c(x^2_i) \neq 0] w^-(O^2) + \sum_{i \notin O^2} \mathbbm{1}[c(x^2_i) \neq 1] w^+(O^2) \right) \\
\end{aligned}
\end{equation}
The following lemma reduces both the number of unlabeled and positive samples in PU learning over $\mathcal{W}^{agno}_{\rho, B}$ to the generalized weighted die problem.
\begin{lemma} \label{lem:agnostic-PU-to-weighted-die-unlabel}
Let $\varepsilon, \delta, \rho \in (0, 1)$ and $m, n \in \Nat$. Suppose there exists a PU learner $\mathcal{A}$ such that for all $(\D, \D_+) \in \mathcal{W}^{agno}_{\rho, B}$, $\nP \geq m$, $\nU \geq n$ satisfies
$$
Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} [\err_\D(\mathcal{A}(S^P, S^U)) - \min_{\text{functions }f} \err_\D(f) \geq \varepsilon] \leq \delta
$$
Then, (i) there exists a learner that with probability $1 - \delta$ achieves error less than $\frac{4\varepsilon}{\rho}$ using $m$ rolls for a weighted generalized with parameters $k$ and $\rho$; (ii) there exists a learner that with probability $1 - \delta$ achieves error less than $\frac{4\varepsilon}{\rho}$ using $n$ rolls for a weighted generalized with parameters $k$ and $\frac{\rho}{2}$.
\end{lemma}
\begin{proof}
We prove the first part, and the second part is identical to the first part. Fix any die corresponding to $O \in V_k$. Suppose $S = \{i_1, i_2, ..., i_m\}$ is a roll of size $m$ from $P_O$. Let $O' = \{1\}$. First notice that for any $x \in \X$ we have
$$
\D_{+, O, O'} := \begin{cases}
\frac{1}{2k} & x \in B^1 \\
\frac{1 + \frac{ \rho}{2 (k - 1)} }{2 k} & \text{if } x = x^2_i \text{ and } i \neq 1 \\
\frac{1 - \frac{ \rho}{2} }{2 k} & \text{if } x = x^2_1 \\
0 & \text{o.w.}
\end{cases}
$$
Moreover,
\begin{equation} \label{eq:D-marg-B2}
\D_{\X, O, O'}(x \mid x \in B^2) =
\begin{cases}
\frac{1 + \frac{\rho}{k - 1}}{k} & \text{if } x = x^2_i \text{ and } i \neq 1\\
\frac{1 - \rho}{k} & \text{if } x = x^2_1 \\
0 & \text{o.w.}
\end{cases}
\end{equation}
Notice that both $\D_{+, O, O'}$ and $\D_{\X, O, O'}(. \mid x \in B^2)$ are independent from $O$, and thus a learner can collect samples from them without requiring knowledge about $O$. Next, we define $S^P$ to be an i.i.d sample of size $n$ from $\D_{+, O, O'}$. Then, construct $S^U$ as follows. Initialize $j = 0$. At each step, flip an unbiased coin. If it lands heads, increment $j$ by 1 and add $x^1_{i_j}$ to $S^U$. If it lands tails, draw a sample from $\D_{\X, O, O'}(\cdot \mid x \in B^2)$ and add it to $S^U$. Repeat this process until $j = m$.
Moreover, observe that $\D_{\X, O, O'}(\cdot \mid x \in B^1) = P_O$. Hence, each $x^1_{i_j}$ is an independent sample from $\D_{\X, O, O'}(\cdot \mid x \in B^1)$. Since $\D_{\X, O, O'}(B^1) = \frac{1}{2}$, it follows that $S^U$ is an i.i.d. sample from $\D_{\X, O, O'}$. Note that due to lemma's assumption, since $|S^U| \geq m$, $|S^P| \geq n$ with probability $1 - \delta$ we have
$$
\err_{\D_{O, O'}}(\mathcal{A}(S^P, S^U)) - \min_{\text{functions }f} \err_{\D_{O, O'}}(f) < \varepsilon
$$
We define our leaner $h \in \{0, 1\}^k$ as $h_i := (1 - \mathcal{A}(S^P, S^U)(x^1_i))$ for $i \in [k]$. Due to \eqref{eq:excess-risk} we have
$$
\begin{aligned}
& \err_{\D_{O, O'}}\left(\mathcal{A}(S^P, S^U)\right) - \min_{\text{functions }f} \err_{\D_{O, O'}}(f) \\
& \geq \frac{\rho}{4k} \left( \sum_{i \in O} \mathbbm{1}\left[\mathcal{A}(S^P, S^U)(x^1_i) \neq 1\right] w^-(O) + \sum_{i \notin O} \mathbbm{1} \left[\mathcal{A}(S^P, S^U)(x^1_i) \neq 0\right] w^+(O) \right) \\
& = \frac{\rho}{4} \err(h)
\end{aligned}
$$
Thus $\err(h) \leq \frac{4 \varepsilon}{\rho}$ with probability $1 - \delta$.
\end{proof}
Then, we first show that for $d \geq 64$ we have $m^{unlabel}_\C(\varepsilon, \frac{1}{320}) \geq \lfloor \frac{d}{2} \rfloor \lfloor \frac{1}{819200 \varepsilon^2} \rfloor, m^{pos}_\C(\varepsilon, \frac{1}{320}) \geq \lfloor \frac{d}{2} \rfloor \lfloor \frac{1}{204800 \varepsilon^2} \rfloor$. Let $\varepsilon < \frac{1}{640}$, and set $k = \lfloor \frac{d}{2} \rfloor$. Let $B$ be any subset of $\X$ of size $2k$ that is shattered by $\C$. Define $\rho = 640 \varepsilon$, and let the number of unlabeled examples be $\nU = k \left\lfloor \frac{1}{2 \rho^2} \right\rfloor$, while the number of positive examples is any $\nP \in \Nat$. For the sake of contradiction, suppose there exists a PU learner $\mathcal{A}$ such that for all $(\D, \D_+) \in \mathcal{W}^{agno}_{\rho, B}$ satisfies
$$
Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} [\err_\D(\mathcal{A}(S^P, S^U)) - \min_{c \in \C} \err_\D(\mathcal{A}(S^P, S^U)) \geq \varepsilon] \leq \frac{1}{320}
$$
Using Lemma~\ref{lem:agnostic-PU-to-weighted-die-unlabel} generalized weighted die problem with parameters $k$ and $\rho$ can achieve error $\frac{1}{160}$ with probability $\frac{1}{320}$ with $\nU$ rolls. Assuming $k \geq 32$ (which holds as long as $d \geq 64$) this contradicts Theorem~\ref{thm:agnostic-PU-lower-bound}. Therefore, we can conclude that no matter how many positive samples the learner receives, the sample complexity of unlabeled examples should be at least $m^{unlabel}_\C(\varepsilon, \frac{1}{320}) \geq \lfloor \frac{d}{2} \rfloor \lfloor \frac{1}{819200 \varepsilon^2} \rfloor$. Similarly, we can see that no matter how many unlabeled samples the learner receives, as long as $d \geq 64$, the sample complexity of unlabeled examples should be at least $m^{pos}_\C(\varepsilon, \frac{1}{320}) \geq \lfloor \frac{d}{2} \rfloor \lfloor \frac{1}{204800 \varepsilon^2} \rfloor$.
Finally, we show that for $d \geq 4$, we have $m^{unlabel}_\C(\varepsilon, \delta) \geq \frac{1}{256 \varepsilon^2} \ln\left( \frac{1}{4 \delta}\right), m^{pos}_\C(\varepsilon, \delta) \geq \frac{1}{64 \varepsilon^2} \ln\left( \frac{1}{4 \delta}\right)$. For $\varepsilon < \frac{1}{16}$ and $\delta < \frac{1}{4}$, let $k = 2$ and let $B$ be any subset of size $4$ shattered by $\C$. Define $\rho = 8\varepsilon$, and let the number of unlabeled examples be $\nU = \frac{1}{4\rho^2} \ln\left( \frac{1}{4\delta} \right)$, while the number of positive examples is any $\nP \in \Nat$. For the sake of contradiction, suppose there exists a PU learner $\mathcal{A}$ such that for all $(\D, \D_+) \in \mathcal{W}^{agno}_{\rho, B}$ satisfies
$$
Pr_{S^P \sim \D_+^{\nP}, S^U \sim \D_\X^{\nU}} [\err_\D(\mathcal{A}(S^P, S^U)) - \min_{c \in \C} \err_\D(\mathcal{A}(S^P, S^U)) \geq \varepsilon] \leq \delta
$$
Then, by Lemma~\ref{lem:agnostic-PU-to-weighted-die-unlabel}, the generalized weighted die problem with parameters $2$ and $\rho$ would achieve error less than $\frac{1}{2}$. Since $\rho < \frac{1}{2}$ and $\delta < \frac{1}{4}$, this contradicts Corollary~\ref{lem:ber-sample}. Therefore, we can conclude that no matter how many positive samples the learner receives, the sample complexity of unlabeled examples should be at least $m^{unlabel}_\C(\varepsilon, \delta) \geq \frac{1}{256 \varepsilon^2} \ln\left( \frac{1}{4 \delta}\right)$. Similarly, we can see that no matter how many unlabeled samples the learner receives, as long as $d \geq 4$ the sample complexity of unlabeled examples should be at least $m^{pos}_\C(\varepsilon, \delta) \geq \frac{1}{64 \varepsilon^2} \ln\left( \frac{1}{4 \delta}\right)$.
% \subsection{Proof of Theorem~\ref{thm:PU-agnostic-lag-pos-scar}}
\textbf{Correction:} In our submission, the term $1 - 2\delta$ in Theorems~\ref{thm:PU-agnostic-lag-pos-scar} and \ref{thm:PU-agnostic-lag-pos-arb} was mistakenly written as $1 - 4\delta$.
\thmagnoscarup*
\begin{proof}
Using standard PAC theory, for all $c' \in \C$ there exists some $M > 1$ such that if $|S^U| > \frac{M (d \ln(1 / \varepsilon) + \ln (1 / \delta))}{ \varepsilon^2}$, then with probability $1 - \delta$ we have $\left |\frac{|c' \mid S^U|}{\nU} - Pr(c'(x) = 1) \right| \leq \varepsilon$ (e.g. see \cite{shalev2014understanding}).
% \begin{equation} \label{eq:PU-agnostic-lag-pos-scar-0}
% \begin{aligned}
% Pr\left(\left |\frac{|c' \mid U|}{\nU} - Pr(c'(x) = 1) \right| \geq \varepsilon \right)
% \delta
% \end{aligned}
% \end{equation}
Similarly, there exists some $M > 1$ such that if $|S^P| > \frac{M (d \ln(1 / \varepsilon) + \ln (1 / \delta))}{ \varepsilon^2}$, then with probability $1 - \delta$ we have $\left| \err_{\D}^+(c') - \frac{\nP - |c' \mid S^P|}{\nP} \right| \leq \varepsilon$. Combining these two facts, with probability $1 - 2\delta$ for all $c' \in \C$ we derive
\begin{equation} \label{eq:PU-agnostic-lag-pos-scar-1}
|Pr(c'(x) = 1) + \gamma \err_{\D}^+(c') - \hat \err^\gamma(c') | \leq (1 + \gamma) \varepsilon
\end{equation}
Then, for any $c' \in \C$ we have
\begin{equation} \label{eq:PU-agnostic-lag-pos-scar-2}
\begin{aligned}
\Pr(c'(x) = 1) &= \Pr (y = 1) - \Pr( y = 1, c'(x) = 0) + \Pr(c'(x) = 1, y = 0) \\
&= \prior - \prior \err_{\D}^+(c') + (1 - \prior) \err_{\D}^-(c')
\end{aligned}
\end{equation}
Now, fix any $c \in \C$. For the cases where $\gamma \geq 2\prior$, with probability $1 - 2\delta$ we have
\begin{equation}\label{eq:PU-agnostic-lag-pos-scar-3}
\begin{aligned}
\prior + \err_{\D}(c^{PU}) &= \prior + \prior \err_{\D}^+(c^{PU}) + (1 - \prior) \err_{\D}^-(c^{PU}) \\
& \leq \prior + (\gamma - \prior) \err_{\D}^+(c^{PU}) + (1 - \prior) \err_{\D}^-(c^{PU})\\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-scar-2}}{=} Pr(c^{PU}(x) = 1) + \gamma \err_{\D}^+(c^{PU}) \\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-scar-1}}{\leq} \hat \err^\gamma(c^{PU}) + (1 + \gamma) \varepsilon \\
& \stackrel{(i)}{\leq} \hat \err^\gamma(c) + (1 + \gamma) \varepsilon \\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-scar-1}}{\leq} Pr(c(x) = 1) + \gamma \err_{\D}^+(c) + 2(1 + \gamma) \varepsilon \\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-scar-2}}{=} \prior + (\gamma - \prior) \err_{\D}^+(c) + (1 - \prior) \err_{\D}^-(c) + 2(1 + \gamma) \varepsilon \\
& = \prior + (\gamma - 2\prior) \err_{\D}^+(c) + \err_{\D}(c) + 2(1 + \gamma) \varepsilon \\
& \stackrel{(ii)}{\leq} \prior + \left(1 + \frac{\gamma - 2\prior}{\prior}\right)\err_{\D}(c) + 2(1 + \gamma) \varepsilon.
\end{aligned}
\end{equation}
Where (i) is due to the definition of $c^{PU}$ and (ii) is due to the fact that $\err_{\D}^+(c) \leq \frac{ \err_{\D}(c)}{\prior}$. For the cases where $\gamma < 2 \alpha$ with probability $1 - 2\delta$ we have
\begin{equation}\label{eq:PU-agnostic-lag-pos-scar-4}
\begin{aligned}
\prior + \frac{\gamma - \prior}{\prior} \err_{\D}(c^{PU}) &= \prior + (\gamma - \prior) \err_{\D}^+(c^{PU}) + (1 - \prior) \frac{\gamma - \prior}{\prior} \err_{\D}^-(c^{PU})\\
& \leq
\prior + (\gamma - \prior) \err_{\D}^+(c^{PU}) + (1 - \prior) \err_{\D}^-(c^{PU})\\
& \stackrel{(i)}{\leq} \prior + (\gamma - \prior) \err_{\D}^+(c) + (1 - \prior) \err_{\D}^-(c) + 2(1 + \gamma) \varepsilon \\
& \leq \prior + \err_{\D}(c) + 2(1 + \gamma) \varepsilon
\end{aligned}
\end{equation}
where (i) is derived similarly to \eqref{eq:PU-agnostic-lag-pos-scar-3}. This completes the proof.
\end{proof}
% \subsection{Proof of Theorem~\ref{thm:PU-agnostic-lag-pos-arb}}
% Before proceeding with the proof of Lemma~\ref{lem:epnet-pi}, we begin by presenting the following result by \cite{ben2010theory}.
\begin{theorem}[\cite{ben2010theory}] \label{thm:dom-adapt-ben2006}
Let $\C$ be a VC-dimension $d$ concept class over domain $\X$, and let $\cQ_S$ and $\cQ_T$ be distributions over $\X \times \{0, 1\}$.
% Let $S$ be an i.i.d sample from $\cQ_1$.
Then, with probability at least $1-\delta$, for every $c \in \C$ :
$$
\begin{aligned}
\err_{\cQ_S}(c) \leq \err_{\cQ_T}(c) +d_{\C \triangle \C}\left(\cQ_{\X, S}, \cQ_{\X, T} \right)+\lambda
\end{aligned}
$$
where
% $\hat{\err}_{S}(c) = \frac{1}{|S|} \sum_{(x, y) \in S} \mathbbm{1}\{c(x) \neq y\}$ and
$\lambda := \inf_{c \in \C} \err_{\cQ_S}(c) + \err_{\cQ_T}(c)$.
\end{theorem}
\thmagnoarbup*
\begin{proof}
% \textit{Proof of Theorem~\ref{thm:PU-agnostic-lag-pos-arb}.}
Note that similar to \eqref{eq:PU-agnostic-lag-pos-scar-1}, again we know there exists some $M > 1$ such that for all $c' \in C$ with probability $1 - 2\delta$
\begin{equation} \label{eq:PU-agnostic-lag-pos-sar-1}
|\Pr(c'(x) = 1) + \gamma \err_{\cP}(c', 1) - \hat \err^\gamma(c') | \leq (1 + \gamma) \varepsilon
\end{equation}
Fix any $c \in \C$. Thus, for the cases where $\gamma \geq 2\prior$ similar to \eqref{eq:PU-agnostic-lag-pos-scar-3} with probability $1 - 2 \delta$ we have
\begin{equation}\label{eq:PU-agnostic-lag-pos-sar-2}
\begin{aligned}
\prior + \err_{\D}(c^{PU}) & \stackrel{(i)}{\leq} \Pr(c^{PU}(x) = 1) + \gamma \err_{\D}^+(c^{PU}) \\
& \stackrel{\text{Theorem}~\ref{thm:dom-adapt-ben2006}}{\leq}
\Pr(c^{PU}(x) = 1) + \gamma \err_{\cP}(c^{PU}, 1) + \gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right) \\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-sar-1}}{\leq} \hat \err^\gamma(c^{PU}) + (1 + \gamma) \varepsilon + \gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right)\\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-scar-2}}{\leq} \hat \err^\gamma(c) + (1 + \gamma) \varepsilon + \gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right) \\
& \stackrel{\eqref{eq:PU-agnostic-lag-pos-sar-1} }{\leq} \Pr(c(x) = 1) + \gamma \err_{\cP}(c, 1) + (1 + \gamma) \varepsilon + \gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right)\\
& \stackrel{\text{Theorem}~\ref{thm:dom-adapt-ben2006}}{\leq} \Pr(c(x) = 1) + \gamma \err^+_{\D}(c) + (1 + \gamma) \varepsilon + 2\gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right)\\
& \stackrel{(ii)}{\leq} \prior + \left(1 + \frac{\gamma - 2\prior}{\prior}\right)\err_{\D}(c) + 2(1 + \gamma) \varepsilon + 2\gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right)\\
\end{aligned}
\end{equation}
Where (i) and (ii) are respectively due to the first two lines and last two lines of \eqref{eq:PU-agnostic-lag-pos-scar-3} of Theorem~\ref{thm:PU-agnostic-lag-pos-scar}. Again similar to \eqref{eq:PU-agnostic-lag-pos-scar-4}, in case $\gamma < 2\prior$ with probability $1 - 2\delta$ we have
\begin{equation}\label{eq:PU-agnostic-lag-pos-sar-3}
\begin{aligned}
\prior + \frac{\gamma - \prior}{\prior} \err_{\D}(c^{PU}) & \leq \prior + \err_{\D}(c) + 2(1 + \gamma) \varepsilon + 2\gamma \left( \lambda^P + d_{\C \triangle \C} (\cP, \D_+) \right)
\end{aligned}
\end{equation}
This completes the proof.
\end{proof}
\begin{thebibliography}{}
\bibitem[Anthony and Bartlett, 2009]{anthony2009neural}
Anthony, M. and Bartlett, P.~L. (2009).
\newblock {\em Neural network learning: Theoretical foundations}.
\newblock cambridge university press.
\bibitem[Bekker and Davis, 2018]{bekker2018estimating}
Bekker, J. and Davis, J. (2018).
\newblock Estimating the class prior in positive and unlabeled data through
decision tree induction.
\newblock In {\em Proceedings of the AAAI conference on artificial
intelligence}, volume~32.
\bibitem[Bekker and Davis, 2020]{bekker2020learning}
Bekker, J. and Davis, J. (2020).
\newblock Learning from positive and unlabeled data: A survey.
\newblock {\em Machine Learning}, 109(4):719--760.
\bibitem[Bekker et~al., 2019]{bekker2019beyond}
Bekker, J., Robberechts, P., and Davis, J. (2019).
\newblock Beyond the selected completely at random assumption for learning from
positive and unlabeled data.
\newblock In {\em Joint European conference on machine learning and knowledge
discovery in databases}, pages 71--85. Springer.
\bibitem[Ben-David and Ben-David, 2011]{ben2011learning}
Ben-David, S. and Ben-David, S. (2011).
\newblock Learning a classifier when the labeling is known.
\newblock In {\em International Conference on Algorithmic Learning Theory},
pages 440--451. Springer.
\bibitem[Ben-David et~al., 2010]{ben2010theory}
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan,
J.~W. (2010).
\newblock A theory of learning from different domains.
\newblock {\em Machine learning}, 79:151--175.
\bibitem[Ben-David and Litman, 1998]{ben1998combinatorial}
Ben-David, S. and Litman, A. (1998).
\newblock Combinatorial variability of vapnik-chervonenkis classes with
applications to sample compression schemes.
\newblock {\em Discrete Applied Mathematics}, 86(1):3--25.
\bibitem[Ben-David and Urner, 2012]{ben2012hardness}
Ben-David, S. and Urner, R. (2012).
\newblock On the hardness of domain adaptation and the utility of unlabeled
target samples.
\newblock In {\em Algorithmic Learning Theory: 23rd International Conference,
ALT 2012, Lyon, France, October 29-31, 2012. Proceedings 23}, pages 139--153.
Springer.
\bibitem[Blanchard et~al., 2010]{blanchard2010semi}
Blanchard, G., Lee, G., and Scott, C. (2010).
\newblock Semi-supervised novelty detection.
\newblock {\em The Journal of Machine Learning Research}, 11:2973--3009.
\bibitem[Christoffel et~al., 2016]{pmlr-v45-Christoffel15}
Christoffel, M., Niu, G., and Sugiyama, M. (2016).
\newblock Class-prior estimation for learning from positive and unlabeled data.
\newblock In Holmes, G. and Liu, T.-Y., editors, {\em Asian Conference on
Machine Learning}, volume~45 of {\em Proceedings of Machine Learning
Research}, pages 221--236, Hong Kong. PMLR.
\bibitem[Coudray et~al., 2023]{coudray2023risk}
Coudray, O., Keribin, C., Massart, P., and Pamphile, P. (2023).
\newblock Risk bounds for positive-unlabeled learning under the selected at
random assumption.
\newblock {\em Journal of Machine Learning Research}, 24(107):1--31.
\bibitem[Dai et~al., 2023]{dai2023gradpu}
Dai, S., Li, X., Zhou, Y., Ye, X., and Liu, T. (2023).
\newblock Gradpu: positive-unlabeled learning via gradient penalty and positive
upweighting.
\newblock In {\em Proceedings of the AAAI conference on artificial
intelligence}, volume~37, pages 7296--7303.
\bibitem[Du~Plessis et~al., 2015]{du2015convex}
Du~Plessis, M., Niu, G., and Sugiyama, M. (2015).
\newblock Convex formulation for learning from positive and unlabeled data.
\newblock In {\em International conference on machine learning}, pages
1386--1394. PMLR.
\bibitem[Du~Plessis and Sugiyama, 2014]{du2014class}
Du~Plessis, M.~C. and Sugiyama, M. (2014).
\newblock Class prior estimation from positive and unlabeled data.
\newblock {\em IEICE TRANSACTIONS on Information and Systems},
97(5):1358--1362.
\bibitem[Elkan and Noto, 2008]{elkan2008learning}
Elkan, C. and Noto, K. (2008).
\newblock Learning classifiers from only positive and unlabeled data.
\newblock In {\em Proceedings of the 14th ACM SIGKDD international conference
on Knowledge discovery and data mining}, pages 213--220.
\bibitem[Feller, 1991]{feller1991introduction}
Feller, W. (1991).
\newblock {\em An introduction to probability theory and its applications,
Volume 2}, volume~81.
\newblock John Wiley \& Sons.
\bibitem[Gong et~al., 2021]{gong2021instance}
Gong, C., Wang, Q., Liu, T., Han, B., You, J., Yang, J., and Tao, D. (2021).
\newblock Instance-dependent positive and unlabeled learning with labeling bias
estimation.
\newblock {\em IEEE transactions on pattern analysis and machine intelligence},
44(8):4163--4177.
\bibitem[Hammoudeh and Lowd, 2020]{hammoudeh2020learning}
Hammoudeh, Z. and Lowd, D. (2020).
\newblock Learning from positive and unlabeled data with arbitrary positive
shift.
\newblock {\em Advances in Neural Information Processing Systems},
33:13088--13099.
\bibitem[Haussler and Welzl, 1987]{hausslerwelzl1987}
Haussler, D. and Welzl, E. (1987).
\newblock epsilon-nets and simplex range queries.
\newblock {\em Discrete \& Computational Geometry}, 2(2):127--151.
\bibitem[He et~al., 2018]{he2018instance}
He, F., Liu, T., Webb, G.~I., and Tao, D. (2018).
\newblock Instance-dependent pu learning by bayesian optimal relabeling.
\newblock {\em arXiv preprint arXiv:1808.02180}.
\bibitem[Hoeffding, 1994]{hoeffding1994probability}
Hoeffding, W. (1994).
\newblock Probability inequalities for sums of bounded random variables.
\newblock {\em The collected works of Wassily Hoeffding}, pages 409--426.
\bibitem[Jain et~al., 2016]{jain2016estimating}
Jain, S., White, M., and Radivojac, P. (2016).
\newblock Estimating the class prior and posterior from noisy positives and
unlabeled data.
\newblock {\em Advances in neural information processing systems}, 29.
\bibitem[Kato et~al., 2019]{kato2019learning}
Kato, M., Teshima, T., and Honda, J. (2019).
\newblock Learning from positive and unlabeled data with a selection bias.
\newblock In {\em International conference on learning representations}.
\bibitem[Kelly et~al., 2010]{kelly2010universal}
Kelly, B.~G., Tularak, T., Wagner, A.~B., and Viswanath, P. (2010).
\newblock Universal hypothesis testing in the learning-limited regime.
\newblock In {\em 2010 IEEE International Symposium on Information Theory},
pages 1478--1482. IEEE.
\bibitem[Kifer et~al., 2004]{kifer2004detecting}
Kifer, D., Ben-David, S., and Gehrke, J. (2004).
\newblock Detecting change in data streams.
\newblock In {\em VLDB}, volume~4, pages 180--191. Toronto, Canada.
\bibitem[Kumar and Lambert, 2023]{kumar2023positive}
Kumar, P. and Lambert, C.~G. (2023).
\newblock Positive unlabeled learning selected not at random (pulsnar): class
proportion estimation when the scar assumption does not hold.
\newblock {\em arXiv preprint arXiv:2303.08269}.
\bibitem[Lee et~al., 2025]{lee2025learning}
Lee, J.~H., Mehrotra, A., and Zampetakis, M. (2025).
\newblock Learning with positive and imperfect unlabeled data.
\newblock {\em arXiv preprint arXiv:2504.10428}.
\bibitem[Li et~al., 2010]{li2010positive}
Li, W., Guo, Q., and Elkan, C. (2010).
\newblock A positive and unlabeled learning algorithm for one-class
classification of remote-sensing data.
\newblock {\em IEEE transactions on geoscience and remote sensing},
49(2):717--725.
\bibitem[Liu et~al., 2002]{liu2002partially}
Liu, B., Lee, W.~S., Yu, P.~S., and Li, X. (2002).
\newblock Partially supervised classification of text documents.
\newblock In {\em ICML}, volume~2, pages 387--394. Sydney, NSW.
\bibitem[Liu and Tao, 2015]{liu2015classification}
Liu, T. and Tao, D. (2015).
\newblock Classification with noisy labels by importance reweighting.
\newblock {\em IEEE Transactions on pattern analysis and machine intelligence},
38(3):447--461.
\bibitem[Motwani and Raghavan, 1996]{motwani1996randomized}
Motwani, R. and Raghavan, P. (1996).
\newblock Randomized algorithms.
\newblock {\em ACM Computing Surveys (CSUR)}, 28(1):33--37.
\bibitem[Na et~al., 2020]{na2020deep}
Na, B., Kim, H., Song, K., Joo, W., Kim, Y.-Y., and Moon, I.-C. (2020).
\newblock Deep generative positive-unlabeled learning under selection bias.
\newblock In {\em Proceedings of the 29th acm international conference on
information \& knowledge management}, pages 1155--1164.
\bibitem[Ramaswamy et~al., 2016]{ramaswamy2016mixture}
Ramaswamy, H., Scott, C., and Tewari, A. (2016).
\newblock Mixture proportion estimation via kernel embeddings of distributions.
\newblock In {\em International conference on machine learning}, pages
2052--2060. PMLR.
\bibitem[Sakai and Shimizu, 2019]{sakai2019covariate}
Sakai, T. and Shimizu, N. (2019).
\newblock Covariate shift adaptation on learning from positive and unlabeled
data.
\newblock In {\em Proceedings of the AAAI conference on artificial
intelligence}, volume~33, pages 4838--4845.
\bibitem[Scott, 2015]{scott2015rate}
Scott, C. (2015).
\newblock A rate of convergence for mixture proportion estimation, with
application to learning from noisy labels.
\newblock In {\em Artificial Intelligence and Statistics}, pages 838--846.
PMLR.
\bibitem[Shalev-Shwartz and Ben-David, 2014]{shalev2014understanding}
Shalev-Shwartz, S. and Ben-David, S. (2014).
\newblock {\em Understanding machine learning: From theory to algorithms}.
\newblock Cambridge university press.
\bibitem[Slud, 1977]{slud1977distribution}
Slud, E.~V. (1977).
\newblock Distribution inequalities for the binomial law.
\newblock {\em The Annals of Probability}, 5(3):404--412.
\bibitem[Tate, 1953]{tate1953double}
Tate, R.~F. (1953).
\newblock On a double inequality of the normal distribution.
\newblock {\em The Annals of Mathematical Statistics}, 24(1):132--134.
\bibitem[Yang et~al., 2012]{yang2012positive}
Yang, P., Li, X.-L., Mei, J.-P., Kwoh, C.-K., and Ng, S.-K. (2012).
\newblock Positive-unlabeled learning for disease gene identification.
\newblock {\em Bioinformatics}, 28(20):2640--2647.
\end{thebibliography}