Citation Context

...ement. Then with probability at least 3/4 over random samples U , the following holds uniformly for all candidate subspace sets C: cost(C;XS) ≤ 2cost(C;X ) + γ. (13) B.1 Proof of Lemma B.1 Lemma B.2 (=-=[39]-=-). Fix X and f : X → [0,M ] for some positive constant M > 0. Let XS be a subset of X with t elements, each drawn uniformly at random from X without replacement. Let , δ > 0. Then Pr[|EX [f(x)]− EXS ...

Citation Context

... absolute constants c1 > 0, 0 < c2 < 1 such that for every t > 0, Pr [ σn(A) ≤ t √ n ] ≤ c1t+ cn2 , where σn(A) is the least singular value of A. Theorem F.2 (Wedin’s theorem; Theorem 4.1, pp. 260 in =-=[45]-=-). Let A,E ∈ Rm×n be given matrices with m ≥ n. Let A have the following singular value decomposition U>1U>2 U>3 A [ V1 V2 ] = [ Σ1 00 Σ2 0 0 ] , 20 where U1,U2,U3,V1,V2 have orthonormal columns a...

Citation Context

... privacy parameter ε is truly large (which means very little privacy protection). On the other hand, both Gibbs sampling and SuLQ subspace clustering give reasonably good performance. Figure 2 also shows that SuLQ scales poorly with the ambient dimension d. This is because SuLQ subspace clustering requires calibrating noise to a d × d sample covariance matrix, which induces much error when d is large. Gibbs sampling seems to be robust to various d settings. We also experiment on real-world datasets. The right two plots in Figure 2 report utility on a subset of the extended Yale Face Dataset B [13] for face clustering. 5 random individuals are picked, forming a subset of the original dataset with n = 320 data points (images). The dataset is preprocessed by projecting each individual onto a 9D affine subspace via PCA. Such preprocessing step was adopted in [32, 29] and was theoretically justified in [1]. Afterwards, ambient dimension of the entire dataset is reduced to d = 50 by random Gaussian projection. The plots show that Gibbs sampling significantly outperforms the other algorithms. 4In MPPCA latent variables yi are sampled from a normal distributionN (0, ρ2Iq). 7 −1 −0.5 0 0.5 1 1....

Citation Context

... that if utility is measured in terms of exact clustering, then no private subspace clustering algorithm exists when neighboring databases are allowed to differ on an entire user profile. In addition, state-of-the-art subspace clustering methods like Sparse Subspace Clustering (SSC, [11]) lack a complete analysis of its clustering output, thanks to the notorious “graph connectivity” problem [21]. Finally, clustering could have high global sensitivity even if only cluster centers are released, as depicted in Figure 1. As a result, general private data releasing schemes like output perturbation [7, 8, 2] do not apply. In this work, we present a systematic and principled treatment of differentially private subspace clustering. To circumvent the negative result in [29], we use the perturbation of recovered low1 dimensional subspace from the ground truth as the utility measure. Our contributions are two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framework [22] and established formal privacy and utility guarantees when data are generated from some stochastic model or satisfy certain deterministic separation conditions. New results on (non-private) subspace clus...

Citation Context

...ibrating noise to a d × d sample covariance matrix, which induces much error when d is large. Gibbs sampling seems to be robust to various d settings. We also experiment on real-world datasets. The right two plots in Figure 2 report utility on a subset of the extended Yale Face Dataset B [13] for face clustering. 5 random individuals are picked, forming a subset of the original dataset with n = 320 data points (images). The dataset is preprocessed by projecting each individual onto a 9D affine subspace via PCA. Such preprocessing step was adopted in [32, 29] and was theoretically justified in [1]. Afterwards, ambient dimension of the entire dataset is reduced to d = 50 by random Gaussian projection. The plots show that Gibbs sampling significantly outperforms the other algorithms. 4In MPPCA latent variables yi are sampled from a normal distributionN (0, ρ2Iq). 7 −1 −0.5 0 0.5 1 1.5 2 2.5 3 0 0.05 0.1 0.15 0.2 0.25 0.3 Log 10 ε K − m ea ns c os t s.a., SSC s.a., TSC s.a., LRR exp., SSC exp. LRR SuLQ−10 SuLQ−50 −1 −0.5 0 0.5 1 1.5 2 2.5 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Log 10 ε K − m ea ns c os t s.a., SSC s.a., TSC s.a., LRR exp., SSC exp. LRR SuLQ−10 SuLQ−50 −1 −0.5 0 0.5 1 1.5 2 2.5 3...

Citation Context

... that if utility is measured in terms of exact clustering, then no private subspace clustering algorithm exists when neighboring databases are allowed to differ on an entire user profile. In addition, state-of-the-art subspace clustering methods like Sparse Subspace Clustering (SSC, [11]) lack a complete analysis of its clustering output, thanks to the notorious “graph connectivity” problem [21]. Finally, clustering could have high global sensitivity even if only cluster centers are released, as depicted in Figure 1. As a result, general private data releasing schemes like output perturbation [7, 8, 2] do not apply. In this work, we present a systematic and principled treatment of differentially private subspace clustering. To circumvent the negative result in [29], we use the perturbation of recovered low1 dimensional subspace from the ground truth as the utility measure. Our contributions are two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framework [22] and established formal privacy and utility guarantees when data are generated from some stochastic model or satisfy certain deterministic separation conditions. New results on (non-private) subspace clus...

Citation Context

...e two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framework [22] and established formal privacy and utility guarantees when data are generated from some stochastic model or satisfy certain deterministic separation conditions. New results on (non-private) subspace clustering are obtained along our analysis, including a fully agnostic subspace clustering on well-separated datasets using stability arguments and exact clustering guarantee for thresholding-based subspace clustering (TSC, [14]) in the noisy setting. In addition, we employ the exponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel tweak in sampling from a matrix Bingham distribution. The method works well in practice and we show it is closely related to the well-known mixtures of probabilistic PCA model [27]. Related work Subspace clustering can be thought as a generalization of PCA and k-means clustering. The former aims at finding a single low-dimensional subspace and the latter uses zerodimensional subspaces as cluster centers. There has been extensive research on private PCA [2, 4, 10] and k-means [2, 22, 26]. Perhaps the ...

Citation Context

...inally, clustering could have high global sensitivity even if only cluster centers are released, as depicted in Figure 1. As a result, general private data releasing schemes like output perturbation [7, 8, 2] do not apply. In this work, we present a systematic and principled treatment of differentially private subspace clustering. To circumvent the negative result in [29], we use the perturbation of recovered low1 dimensional subspace from the ground truth as the utility measure. Our contributions are two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framework [22] and established formal privacy and utility guarantees when data are generated from some stochastic model or satisfy certain deterministic separation conditions. New results on (non-private) subspace clustering are obtained along our analysis, including a fully agnostic subspace clustering on well-separated datasets using stability arguments and exact clustering guarantee for thresholding-based subspace clustering (TSC, [14]) in the noisy setting. In addition, we employ the exponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel ...

Citation Context

... that if utility is measured in terms of exact clustering, then no private subspace clustering algorithm exists when neighboring databases are allowed to differ on an entire user profile. In addition, state-of-the-art subspace clustering methods like Sparse Subspace Clustering (SSC, [11]) lack a complete analysis of its clustering output, thanks to the notorious “graph connectivity” problem [21]. Finally, clustering could have high global sensitivity even if only cluster centers are released, as depicted in Figure 1. As a result, general private data releasing schemes like output perturbation [7, 8, 2] do not apply. In this work, we present a systematic and principled treatment of differentially private subspace clustering. To circumvent the negative result in [29], we use the perturbation of recovered low1 dimensional subspace from the ground truth as the utility measure. Our contributions are two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framework [22] and established formal privacy and utility guarantees when data are generated from some stochastic model or satisfy certain deterministic separation conditions. New results on (non-private) subspace clus...

Citation Context

...sults We provide numerical results of both the sample-aggregate and Gibbs sampling algorithms on synthetic and real-world datasets. We also compare with a baseline method implemented based on the k-plane algorithm [3] with perturbed sample covariance matrix via the SuLQ framework [2] (details presented in Appendix E). Three solvers are considered for the sample-aggregate framework: threshold-based subspace clustering (TSC, [14]), which has provable utility guarantee with sampleaggregation on stochastic models, along with sparse subspace clustering (SSC, [11]) and low-rank representation (LRR, [17]), the two state-of-the-art methods for subspace clustering. For Gibbs sampling, we use non-private SSC and LRR solutions as initialization for the Gibbs sampler. All methods are implemented using Matlab. For synthetic datasets, we first generate k random q-dimensional linear subspaces. Each subspace is generated by first sampling a d× q random Gaussian matrix and then recording its column space. n data points are then assigned to one of the k subspaces (clusters) uniformly at random. To generate a data point xi assigned with subspace S`, we first sample yi ∈ Rq with ‖yi‖2 = 1 uniformly at ran...

Citation Context

... E(`) are stochastic, by standard analysis of PCA one can show that the top-q subspace of X(`) converges to the underlying subspace S∗` in probability as the number of data points n` goes to infinity =-=[43]-=-. The theorem then holds because m = o(n) and hence n′ = n/m→∞. C.1 Proof of Lemma 3.5 Proposition C.2. Suppose yi = xi + εi with εi ∼ N (0, σ 2 d Id) and σ > 0. Then with probability at least 1− n2e−...

Citation Context

...rmal privacy and utility guarantees; the other one asymptotically preserves differential privacy while having good performance in practice. Along the course of the proof, we also obtain two new provable guarantees for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests. 1 Introduction Subspace clustering was originally proposed to solve very specific computer vision problems having a union-of-subspace structure in the data, e.g., motion segmentation under an affine camera model [11] or face clustering under Lambertian illumination models [15]. As it gains increasing attention in the statistics and machine learning community, people start to use it as an agnostic learning tool in social network [5], movie recommendation [33] and biological datasets [19]. The growing applicability of subspace clustering in these new domains inevitably raises the concern of data privacy, as many such applications involve dealing with sensitive information. For example, [19] applies subspace clustering to identify diseases from personalized medical data and [33] in fact uses subspace clustering as a effective tool to conduct linkage attacks on individ...

Citation Context

...s with privacy parameters ε and δ. Then the overall algorithm is (ε′, δ′)- differentially private with ε′ = √ 2kT ln(1/δ)ε+ kTε(eε − 1), δ′ = (kT + 1)δ. Appendix F Concentration theorems Theorem F.1 (=-=[44]-=-, Theorem 1.2). Let A be an n × n matrices with entries i.i.d. sampled from standard Gaussian distribution. Then there exist absolute constants c1 > 0, 0 < c2 < 1 such that for every t > 0, Pr [ σn(A)...

Citation Context

...y and experiments that one of the presented methods enjoys formal privacy and utility guarantees; the other one asymptotically preserves differential privacy while having good performance in practice. Along the course of the proof, we also obtain two new provable guarantees for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests. 1 Introduction Subspace clustering was originally proposed to solve very specific computer vision problems having a union-of-subspace structure in the data, e.g., motion segmentation under an affine camera model [11] or face clustering under Lambertian illumination models [15]. As it gains increasing attention in the statistics and machine learning community, people start to use it as an agnostic learning tool in social network [5], movie recommendation [33] and biological datasets [19]. The growing applicability of subspace clustering in these new domains inevitably raises the concern of data privacy, as many such applications involve dealing with sensitive information. For example, [19] applies subspace clustering to identify diseases from personalized medical data and [33] in fact uses subspace cluster...

Citation Context

...ition 3.2 (Well-separation condition for k-means subspace clustering). A dataset X is (φ, η, ψ)-well separated if there exist constants φ, η and ψ, all between 0 and 1, such that ∆2k(X ) ≤ min { φ2∆2k−1(X ),∆2k,−(X )− ψ,∆2k,+(X ) + η } , (6) where ∆k−1, ∆k,− and ∆k,+ are defined as ∆2k−1(X ) = minS1:k−1∈Sdq cost({Si};X ); ∆ 2 k,−(X ) = minS1∈Sdq−1,S2:k∈Sdq cost({Si};X ); and ∆ 2 k,+(X ) = minS1∈Sdq+1,S2:k∈Sdq cost({Si};X ). The first condition in Eq. (6), ∆2k(X ) ≤ φ2∆2k−1(X ), constrains that the input dataset X cannot be well clustered using k − 1 instead of k clusters. It was introduced in [23] to analyze stability of k-means solutions. For subspace clustering, we need another two conditions regarding the intrinsic dimension of each subspace. The ∆2k(X ) ≤ ∆2k,−(X ) − ψ asserts that replacing a q-dimensional subspace with a (q − 1)-dimensional one is not sufficient, while ∆2k(X ) ≤ ∆2k,+(X ) + η means an additional subspace dimension does not help much with clustering X . The following lemma is our main stability result for subspace clustering on well-separated datasets. It states that when a candidate clustering C is close to the optimal clustering C∗ in terms of clustering cost, ...

Citation Context

...`)); U` ∈ Rd×q,U>` U` = Iq×q, (12) where A` = X(`)X(`) > is the unnormalized sample covariance matrix. Distribution of the form in Eq. (12) is a special case of the matrix Bingham distribution, which admits a Gibbs sampler [16]. We give implementation details in Appendix D.2 with modifications so that the resulting Gibbs sampler is empirically more efficient for a wide range of parameter settings. 3Recently [28] established full clustering guarantee for SSC, however, under strong assumptions. 6 4.2 Discussion The proposed Gibbs sampler resembles the k-plane algorithm for subspace clustering [3]. It is in fact a “probabilistic” version of k-plane since sampling is performed at each iteration rather than deterministic updates. Furthermore, the proposed Gibbs sampler could be viewed as posterior sampling for the following generative model: first sample U` uniformly at random from Sdq for each subspace S`; afterwards, cluster assignments {zi}ni=1 are sampled such that Pr[zi = j] = 1/k and xi is set as xi = U`yi + PU⊥` wi, where yi is sampled uniformly at random from the qdimensional unit ball and wi ∼ N (0, Id/ε). Connection between the above-mentioned generative model and Gibbs sampler...

Citation Context

... for agnostic k-means subspace clustering, with additional amount of per-coordinate Gaussian noise upper bounded by O( φ 2 √ k ε(ψ−η) ). Our bound is comparable to the one obtained in [22] for private k-means clustering, except for the (ψ−η) term which characterizes the well-separatedness under the subspace clustering scenario. 3.3 The stochastic setting We further consider the case when data points are stochastically generated from some underlying “true” subspace set C∗ = {S∗1 , · · · ,S∗k}. Such settings were extensively investigated in previous development of subspace clustering algorithms [24, 25, 14]. Below we give precise definition of the considered stochastic subspace clustering model: The stochastic model For every cluster ` associated with subspace S∗` , a data point x (`) i ∈ Rd belonging to cluster ` can be written as x(`)i = y (`) i + ε (`) i , where y (`) i is sampled uniformly at random from {y ∈ S∗` : ‖y‖2 = 1} and εi ∼ N (0, σ2/d · Id) for some noise parameter σ. Under the stochastic setting we consider the solver f to be the Threshold-based Subspace Clustering (TSC, [14]) algorithm. A simplified version of TSC is presented in Alg. 2. An alternative idea is to apply results in...

Citation Context

... the proof, we also obtain two new provable guarantees for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests. 1 Introduction Subspace clustering was originally proposed to solve very specific computer vision problems having a union-of-subspace structure in the data, e.g., motion segmentation under an affine camera model [11] or face clustering under Lambertian illumination models [15]. As it gains increasing attention in the statistics and machine learning community, people start to use it as an agnostic learning tool in social network [5], movie recommendation [33] and biological datasets [19]. The growing applicability of subspace clustering in these new domains inevitably raises the concern of data privacy, as many such applications involve dealing with sensitive information. For example, [19] applies subspace clustering to identify diseases from personalized medical data and [33] in fact uses subspace clustering as a effective tool to conduct linkage attacks on individuals in movie rating datasets. Nevertheless, privacy issues in subspace clustering have been less explored in the past literature, with the only exception of ...

Citation Context

... for agnostic k-means subspace clustering, with additional amount of per-coordinate Gaussian noise upper bounded by O( φ 2 √ k ε(ψ−η) ). Our bound is comparable to the one obtained in [22] for private k-means clustering, except for the (ψ−η) term which characterizes the well-separatedness under the subspace clustering scenario. 3.3 The stochastic setting We further consider the case when data points are stochastically generated from some underlying “true” subspace set C∗ = {S∗1 , · · · ,S∗k}. Such settings were extensively investigated in previous development of subspace clustering algorithms [24, 25, 14]. Below we give precise definition of the considered stochastic subspace clustering model: The stochastic model For every cluster ` associated with subspace S∗` , a data point x (`) i ∈ Rd belonging to cluster ` can be written as x(`)i = y (`) i + ε (`) i , where y (`) i is sampled uniformly at random from {y ∈ S∗` : ‖y‖2 = 1} and εi ∼ N (0, σ2/d · Id) for some noise parameter σ. Under the stochastic setting we consider the solver f to be the Threshold-based Subspace Clustering (TSC, [14]) algorithm. A simplified version of TSC is presented in Alg. 2. An alternative idea is to apply results in...

Citation Context

... to prove stability results as outlined in Eq. (5) for particular subspace clustering solvers and then apply Theorem 3.1. 3.2 The agnostic setting We first consider the setting when data points {xi}ni=1 are arbitrarily placed. Under such agnostic setting the optimal solution C∗ is defined as the one that minimizes the k-means cost as in Eq. (3). The solver f is taken to be any (1 + )-approximation2 of optimal k-means subspace clustering; that is, f always outputs subspaces C satisfying cost(C;X ) ≤ (1 + )cost(C∗;X ). Efficient core-set based approximation algorithms exist, for example, in [12]. The key task of this section it to identify assumptions under which the stability condition in Eq. (5) holds with respect to an approximate solver f . The example given in Figure 1 also suggests that identifiability issue arises when the input data X itself cannot be well clustered. For example, no two straight lines could well approximate data uniformly distributed on a circle. To circumvent the above-mentioned difficulty, we impose the following well-separation condition on the input data X : Definition 3.2 (Well-separation condition for k-means subspace clustering). A dataset X is (φ, η, ...

Citation Context

...)(q − 1) ≤ 0.02. (17) The second inequality is an application of Eq. (5.2) in [42] and the last inequality is due to Eq. (16). On the other hand, by tail bounds of binomial distribution (Theorem 1 in =-=[41]-=-) we have Pr [∣∣C(a(`)i , rθ∗)∣∣ > n`(p+ 0.01)] ≤ e− 0.012n2`2(pn`+0.01n`/3) ≤ e− n`400 , (18) where in the last inequality we used the fact that pn` ≤ 0.01n`. Since s̄ ≤ 0.02n` ≤ s̃, we proved that w...

Citation Context

... (non-private) subspace clustering are obtained along our analysis, including a fully agnostic subspace clustering on well-separated datasets using stability arguments and exact clustering guarantee for thresholding-based subspace clustering (TSC, [14]) in the noisy setting. In addition, we employ the exponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel tweak in sampling from a matrix Bingham distribution. The method works well in practice and we show it is closely related to the well-known mixtures of probabilistic PCA model [27]. Related work Subspace clustering can be thought as a generalization of PCA and k-means clustering. The former aims at finding a single low-dimensional subspace and the latter uses zerodimensional subspaces as cluster centers. There has been extensive research on private PCA [2, 4, 10] and k-means [2, 22, 26]. Perhaps the most similar work to ours is [22, 4]. [22] applies the sample-aggregate framework to k-means clustering and [4] employs the exponential mechanism to recover private principal vectors. In this paper we give non-trivial generalization of both work to the private subspace clust...

Citation Context

...ecause SuLQ subspace clustering requires calibrating noise to a d × d sample covariance matrix, which induces much error when d is large. Gibbs sampling seems to be robust to various d settings. We also experiment on real-world datasets. The right two plots in Figure 2 report utility on a subset of the extended Yale Face Dataset B [13] for face clustering. 5 random individuals are picked, forming a subset of the original dataset with n = 320 data points (images). The dataset is preprocessed by projecting each individual onto a 9D affine subspace via PCA. Such preprocessing step was adopted in [32, 29] and was theoretically justified in [1]. Afterwards, ambient dimension of the entire dataset is reduced to d = 50 by random Gaussian projection. The plots show that Gibbs sampling significantly outperforms the other algorithms. 4In MPPCA latent variables yi are sampled from a normal distributionN (0, ρ2Iq). 7 −1 −0.5 0 0.5 1 1.5 2 2.5 3 0 0.05 0.1 0.15 0.2 0.25 0.3 Log 10 ε K − m ea ns c os t s.a., SSC s.a., TSC s.a., LRR exp., SSC exp. LRR SuLQ−10 SuLQ−50 −1 −0.5 0 0.5 1 1.5 2 2.5 3 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 Log 10 ε K − m ea ns c os t s.a., SSC s.a., TSC s.a., LRR exp., SSC exp. LRR SuLQ...

Citation Context

...sufficient side information. It is perhaps reasonable why there is little work focusing on private subspace clustering, which is by all means a challenging task. For example, a negative result in [29] shows that if utility is measured in terms of exact clustering, then no private subspace clustering algorithm exists when neighboring databases are allowed to differ on an entire user profile. In addition, state-of-the-art subspace clustering methods like Sparse Subspace Clustering (SSC, [11]) lack a complete analysis of its clustering output, thanks to the notorious “graph connectivity” problem [21]. Finally, clustering could have high global sensitivity even if only cluster centers are released, as depicted in Figure 1. As a result, general private data releasing schemes like output perturbation [7, 8, 2] do not apply. In this work, we present a systematic and principled treatment of differentially private subspace clustering. To circumvent the negative result in [29], we use the perturbation of recovered low1 dimensional subspace from the ground truth as the utility measure. Our contributions are two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framewo...

Citation Context

...l subspace from the ground truth as the utility measure. Our contributions are two-fold. First, we analyze two efficient algorithms based on the sample-aggregate framework [22] and established formal privacy and utility guarantees when data are generated from some stochastic model or satisfy certain deterministic separation conditions. New results on (non-private) subspace clustering are obtained along our analysis, including a fully agnostic subspace clustering on well-separated datasets using stability arguments and exact clustering guarantee for thresholding-based subspace clustering (TSC, [14]) in the noisy setting. In addition, we employ the exponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel tweak in sampling from a matrix Bingham distribution. The method works well in practice and we show it is closely related to the well-known mixtures of probabilistic PCA model [27]. Related work Subspace clustering can be thought as a generalization of PCA and k-means clustering. The former aims at finding a single low-dimensional subspace and the latter uses zerodimensional subspaces as cluster centers. There has been exten...

Citation Context

...roof is then completed by noting that∣∣〈yi,yj〉 − 〈xi,xj〉∣∣ ≤ ∣∣〈εi,xj〉∣∣+ ∣∣〈εj ,xi〉∣∣+ ∣∣〈εi, εj〉∣∣ ≤ (2 √ 5σ + 5σ2) √ 6 log n d . Lemma C.3 ([14], Lemma 3; extracted from the proof of Lemma 6.2. in =-=[42]-=-). Let Sd−1 = {x ∈ Rd : ‖x‖2 = 1} denote the d-dimensional unit sphere. For an arbitrary p ∈ Sd−1, defineC(p, θ) = {q ∈ Sd−1 : ϑ(p, q) ≤ θ} where ϑ(p, q) = arccos(〈p, q〉) is the angle between p and q....

Citation Context

...for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests. 1 Introduction Subspace clustering was originally proposed to solve very specific computer vision problems having a union-of-subspace structure in the data, e.g., motion segmentation under an affine camera model [11] or face clustering under Lambertian illumination models [15]. As it gains increasing attention in the statistics and machine learning community, people start to use it as an agnostic learning tool in social network [5], movie recommendation [33] and biological datasets [19]. The growing applicability of subspace clustering in these new domains inevitably raises the concern of data privacy, as many such applications involve dealing with sensitive information. For example, [19] applies subspace clustering to identify diseases from personalized medical data and [33] in fact uses subspace clustering as a effective tool to conduct linkage attacks on individuals in movie rating datasets. Nevertheless, privacy issues in subspace clustering have been less explored in the past literature, with the only exception of a brief analysis and discussion in [29]. However, the al...

Citation Context

... two new provable guarantees for the agnostic subspace clustering and the graph connectivity problem which might be of independent interests. 1 Introduction Subspace clustering was originally proposed to solve very specific computer vision problems having a union-of-subspace structure in the data, e.g., motion segmentation under an affine camera model [11] or face clustering under Lambertian illumination models [15]. As it gains increasing attention in the statistics and machine learning community, people start to use it as an agnostic learning tool in social network [5], movie recommendation [33] and biological datasets [19]. The growing applicability of subspace clustering in these new domains inevitably raises the concern of data privacy, as many such applications involve dealing with sensitive information. For example, [19] applies subspace clustering to identify diseases from personalized medical data and [33] in fact uses subspace clustering as a effective tool to conduct linkage attacks on individuals in movie rating datasets. Nevertheless, privacy issues in subspace clustering have been less explored in the past literature, with the only exception of a brief analysis and discus...

Citation Context

... sampled uniformly at random from the qdimensional unit ball and wi ∼ N (0, Id/ε). Connection between the above-mentioned generative model and Gibbs sampler is formally justified in Appendix D.3. The generative model is strikingly similar to the well-known mixtures of probabilistic PCA (MPPCA, [27]) model by setting variance parameters σ` in MPPCA to √ 1/ε. The only difference is that yi are sampled uniformly at random from a unit ball 4 and noisewi is constrained to U⊥` , the complement space of U`. Note that this is closely related to earlier observation that “posterior sampling is private” [20, 6, 31], but different in that we constructed a model from a private procedure rather than the other way round. As the privacy parameter ε → ∞ (i.e., no privacy guarantee), we arrive immediately at the exact k-plane algorithm and the posterior distribution concentrates around the optimal k-means solution (C∗, z∗). This behavior is similar to what a small-variance asymptotic analysis on MPPCA models reveals [30]. On the other hand, the proposed Gibbs sampler is significantly different from previous Bayesian probabilisitic PCA formulation [34, 30] in that the subspaces are sampled from a matrix Bingham...

Citation Context

...dition, we employ the exponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel tweak in sampling from a matrix Bingham distribution. The method works well in practice and we show it is closely related to the well-known mixtures of probabilistic PCA model [27]. Related work Subspace clustering can be thought as a generalization of PCA and k-means clustering. The former aims at finding a single low-dimensional subspace and the latter uses zerodimensional subspaces as cluster centers. There has been extensive research on private PCA [2, 4, 10] and k-means [2, 22, 26]. Perhaps the most similar work to ours is [22, 4]. [22] applies the sample-aggregate framework to k-means clustering and [4] employs the exponential mechanism to recover private principal vectors. In this paper we give non-trivial generalization of both work to the private subspace clustering setting. 2 Preliminaries 2.1 Notations For a vector x ∈ Rd, its p-norm is defined as ‖x‖p = ( ∑ i x p i ) 1/p. If p is not explicitly specified then the 2-norm is used. For a matrix A ∈ Rn×m, we use σ1(A) ≥ · · · ≥ σn(A) ≥ 0 to denote its singular values (assuming without loss of ...

Citation Context

...ier observation that “posterior sampling is private” [20, 6, 31], but different in that we constructed a model from a private procedure rather than the other way round. As the privacy parameter ε → ∞ (i.e., no privacy guarantee), we arrive immediately at the exact k-plane algorithm and the posterior distribution concentrates around the optimal k-means solution (C∗, z∗). This behavior is similar to what a small-variance asymptotic analysis on MPPCA models reveals [30]. On the other hand, the proposed Gibbs sampler is significantly different from previous Bayesian probabilisitic PCA formulation [34, 30] in that the subspaces are sampled from a matrix Bingham distribution. Finally, we remark that the proposed Gibbs sampler is only asymptotically private because Proposition 4.1 requires exact (or nearly exact [31]) sampling from Eq. (10). 5 Numerical results We provide numerical results of both the sample-aggregate and Gibbs sampling algorithms on synthetic and real-world datasets. We also compare with a baseline method implemented based on the k-plane algorithm [3] with perturbed sample covariance matrix via the SuLQ framework [2] (details presented in Appendix E). Three solvers are considere...

Citation Context

...dition, we employ the exponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel tweak in sampling from a matrix Bingham distribution. The method works well in practice and we show it is closely related to the well-known mixtures of probabilistic PCA model [27]. Related work Subspace clustering can be thought as a generalization of PCA and k-means clustering. The former aims at finding a single low-dimensional subspace and the latter uses zerodimensional subspaces as cluster centers. There has been extensive research on private PCA [2, 4, 10] and k-means [2, 22, 26]. Perhaps the most similar work to ours is [22, 4]. [22] applies the sample-aggregate framework to k-means clustering and [4] employs the exponential mechanism to recover private principal vectors. In this paper we give non-trivial generalization of both work to the private subspace clustering setting. 2 Preliminaries 2.1 Notations For a vector x ∈ Rd, its p-norm is defined as ‖x‖p = ( ∑ i x p i ) 1/p. If p is not explicitly specified then the 2-norm is used. For a matrix A ∈ Rn×m, we use σ1(A) ≥ · · · ≥ σn(A) ≥ 0 to denote its singular values (assuming without loss of ...

Citation Context

... sampled uniformly at random from the qdimensional unit ball and wi ∼ N (0, Id/ε). Connection between the above-mentioned generative model and Gibbs sampler is formally justified in Appendix D.3. The generative model is strikingly similar to the well-known mixtures of probabilistic PCA (MPPCA, [27]) model by setting variance parameters σ` in MPPCA to √ 1/ε. The only difference is that yi are sampled uniformly at random from a unit ball 4 and noisewi is constrained to U⊥` , the complement space of U`. Note that this is closely related to earlier observation that “posterior sampling is private” [20, 6, 31], but different in that we constructed a model from a private procedure rather than the other way round. As the privacy parameter ε → ∞ (i.e., no privacy guarantee), we arrive immediately at the exact k-plane algorithm and the posterior distribution concentrates around the optimal k-means solution (C∗, z∗). This behavior is similar to what a small-variance asymptotic analysis on MPPCA models reveals [30]. On the other hand, the proposed Gibbs sampler is significantly different from previous Bayesian probabilisitic PCA formulation [34, 30] in that the subspaces are sampled from a matrix Bingham...

Citation Context

...ily done by sampling zj from a categorical distribution. Update of S`: Let X (`) = {xi ∈ X : zi = `} denote data points that are assigned to cluster ` and n` = |X (`)|. Denote X(`) ∈ Rd×n` as the matrix with columns corresponding to all data points in X (`). The distribution over S` conditioned on z can then be written as p(S` = range(U`)|z;X ) ∝ exp(ε/2 · tr(U>` A`U`)); U` ∈ Rd×q,U>` U` = Iq×q, (12) where A` = X(`)X(`) > is the unnormalized sample covariance matrix. Distribution of the form in Eq. (12) is a special case of the matrix Bingham distribution, which admits a Gibbs sampler [16]. We give implementation details in Appendix D.2 with modifications so that the resulting Gibbs sampler is empirically more efficient for a wide range of parameter settings. 3Recently [28] established full clustering guarantee for SSC, however, under strong assumptions. 6 4.2 Discussion The proposed Gibbs sampler resembles the k-plane algorithm for subspace clustering [3]. It is in fact a “probabilistic” version of k-plane since sampling is performed at each iteration rather than deterministic updates. Furthermore, the proposed Gibbs sampler could be viewed as posterior sampling for the follow...

Citation Context

... sampled uniformly at random from the qdimensional unit ball and wi ∼ N (0, Id/ε). Connection between the above-mentioned generative model and Gibbs sampler is formally justified in Appendix D.3. The generative model is strikingly similar to the well-known mixtures of probabilistic PCA (MPPCA, [27]) model by setting variance parameters σ` in MPPCA to √ 1/ε. The only difference is that yi are sampled uniformly at random from a unit ball 4 and noisewi is constrained to U⊥` , the complement space of U`. Note that this is closely related to earlier observation that “posterior sampling is private” [20, 6, 31], but different in that we constructed a model from a private procedure rather than the other way round. As the privacy parameter ε → ∞ (i.e., no privacy guarantee), we arrive immediately at the exact k-plane algorithm and the posterior distribution concentrates around the optimal k-means solution (C∗, z∗). This behavior is similar to what a small-variance asymptotic analysis on MPPCA models reveals [30]. On the other hand, the proposed Gibbs sampler is significantly different from previous Bayesian probabilisitic PCA formulation [34, 30] in that the subspaces are sampled from a matrix Bingham...

Citation Context

...xponential mechanism [18] and propose a novel Gibbs sampler for sampling from this distribution, which involves a novel tweak in sampling from a matrix Bingham distribution. The method works well in practice and we show it is closely related to the well-known mixtures of probabilistic PCA model [27]. Related work Subspace clustering can be thought as a generalization of PCA and k-means clustering. The former aims at finding a single low-dimensional subspace and the latter uses zerodimensional subspaces as cluster centers. There has been extensive research on private PCA [2, 4, 10] and k-means [2, 22, 26]. Perhaps the most similar work to ours is [22, 4]. [22] applies the sample-aggregate framework to k-means clustering and [4] employs the exponential mechanism to recover private principal vectors. In this paper we give non-trivial generalization of both work to the private subspace clustering setting. 2 Preliminaries 2.1 Notations For a vector x ∈ Rd, its p-norm is defined as ‖x‖p = ( ∑ i x p i ) 1/p. If p is not explicitly specified then the 2-norm is used. For a matrix A ∈ Rn×m, we use σ1(A) ≥ · · · ≥ σn(A) ≥ 0 to denote its singular values (assuming without loss of generality that n ≤ m). ...

Citation Context

...d×n` as the matrix with columns corresponding to all data points in X (`). The distribution over S` conditioned on z can then be written as p(S` = range(U`)|z;X ) ∝ exp(ε/2 · tr(U>` A`U`)); U` ∈ Rd×q,U>` U` = Iq×q, (12) where A` = X(`)X(`) > is the unnormalized sample covariance matrix. Distribution of the form in Eq. (12) is a special case of the matrix Bingham distribution, which admits a Gibbs sampler [16]. We give implementation details in Appendix D.2 with modifications so that the resulting Gibbs sampler is empirically more efficient for a wide range of parameter settings. 3Recently [28] established full clustering guarantee for SSC, however, under strong assumptions. 6 4.2 Discussion The proposed Gibbs sampler resembles the k-plane algorithm for subspace clustering [3]. It is in fact a “probabilistic” version of k-plane since sampling is performed at each iteration rather than deterministic updates. Furthermore, the proposed Gibbs sampler could be viewed as posterior sampling for the following generative model: first sample U` uniformly at random from Sdq for each subspace S`; afterwards, cluster assignments {zi}ni=1 are sampled such that Pr[zi = j] = 1/k and xi is set as xi...

Citation Context

...logical datasets [19]. The growing applicability of subspace clustering in these new domains inevitably raises the concern of data privacy, as many such applications involve dealing with sensitive information. For example, [19] applies subspace clustering to identify diseases from personalized medical data and [33] in fact uses subspace clustering as a effective tool to conduct linkage attacks on individuals in movie rating datasets. Nevertheless, privacy issues in subspace clustering have been less explored in the past literature, with the only exception of a brief analysis and discussion in [29]. However, the algorithms and analysis presented in [29] have several notable deficiencies. For example, data points are assumed to be incoherent and it only protects the differential privacy of any feature of a user rather than the entire user profile in the database. The latter means it is possible for an attacker to infer with high confidence whether a particular user is in the database, given sufficient side information. It is perhaps reasonable why there is little work focusing on private subspace clustering, which is by all means a challenging task. For example, a negative result in [29]...

Citation Context

...t random from a unit ball 4 and noisewi is constrained to U⊥` , the complement space of U`. Note that this is closely related to earlier observation that “posterior sampling is private” [20, 6, 31], but different in that we constructed a model from a private procedure rather than the other way round. As the privacy parameter ε → ∞ (i.e., no privacy guarantee), we arrive immediately at the exact k-plane algorithm and the posterior distribution concentrates around the optimal k-means solution (C∗, z∗). This behavior is similar to what a small-variance asymptotic analysis on MPPCA models reveals [30]. On the other hand, the proposed Gibbs sampler is significantly different from previous Bayesian probabilisitic PCA formulation [34, 30] in that the subspaces are sampled from a matrix Bingham distribution. Finally, we remark that the proposed Gibbs sampler is only asymptotically private because Proposition 4.1 requires exact (or nearly exact [31]) sampling from Eq. (10). 5 Numerical results We provide numerical results of both the sample-aggregate and Gibbs sampling algorithms on synthetic and real-world datasets. We also compare with a baseline method implemented based on the k-plane algori...