DMCA
Under consideration for publication in Knowledge and Information Systems Shared-Memory and Shared-Nothing Stochastic Gradient Descent Algorithms for Matrix Completion (2013)
BibTeX
@MISC{Makari13underconsideration,
author = {Faraz Makari and Christina Teflioudi and Rainer Gemulla and Peter Haas and Yannis Sismanis},
title = {Under consideration for publication in Knowledge and Information Systems Shared-Memory and Shared-Nothing Stochastic Gradient Descent Algorithms for Matrix Completion},
year = {2013}
}
Share
 |
 |
 |
 |
||
OpenURL
Abstract
Abstract. We provide parallel algorithms for large-scale matrix completion on problems with millions of rows, millions of columns, and billions of revealed entries. We focus on in-memory algorithms that run either in a shared-memory environment on a powerful compute node or in a shared-nothing environment on a small cluster of commodity nodes; even very large problems can be handled effectively in these settings. Our ASGD, DSGD-MR, DSGD++, and CSGD algorithms are novel variants of the popular stochastic gradient descent (SGD) algorithm, with the latter three algorithms based on a new “stratified SGD ” approach. All of the algorithms are cache-friendly and exploit thread-level parallelism, in-memory processing, and asynchronous communication. We investigate the performance of both new and existing algorithms via a theoretical complexity analysis and a set of large-scale experiments. The results show that CSGD is more scalable, and up to 60 % faster, than the best-performing alternative method in the shared-memory setting. DSGD++ is superior in terms of overall runtime, memory consumption, and scalability in the shared-nothing setting. For example, DSGD++ can solve a difficult matrix completion problem on a high-variance matrix with 10M rows, 1M columns, and 10B revealed entries in around 40 minutes on 16 compute nodes. In general, algorithms based on stochastic gradient descent appear to perform better than algorithms based on alternating minimizations, such as the PALS and DALS alternating least-squares algorithms.
Keyphrases
matrix completion information system shared-memory shared-nothing stochastic gradient descent algorithm revealed entry shared-nothing environment shared-memory setting csgd algorithm in-memory algorithm large-scale matrix completion least-squares algorithm memory consumption exploit thread-level parallelism theoretical complexity analysis asynchronous communication overall runtime high-variance matrix small cluster powerful compute node parallel algorithm popular stochastic gradient descent in-memory processing large-scale experiment stochastic gradient descent new stratified sgd approach compute node shared-nothing setting best-performing alternative method difficult matrix completion problem shared-memory environment novel variant large problem commodity node
