Complexity theory of circuits strongly suggests that deep architectures can be much more efficient (sometimes exponentially) than shallow architectures, in terms of computational elements required to represent some functions. Deep multi-layer neural networks have many levels of non-linearities allowing them to compactly represent highly non-linear and highly-varying functions. However, until recently it was not clear how to train such deep networks, since gradient-based optimization starting from random initialization appears to often get stuck in poor solutions. Hinton et al. recently introduced a greedy layer-wise unsupervised learning algorithm for Deep Belief Networks (DBN), a generative model with many layers of hidden causal variables. In the context of the above optimization problem, we study this algorithm empirically and explore variants to better understand its success and extend it to cases where the inputs are continuous or where the structure of the input distribution is not revealing enough about the variable to be predicted in a supervised task. Our experiments also confirm the hypothesis that the greedy layer-wise unsupervised training strategy mostly helps the optimization, by initializing weights in a region near a good local minimum, giving rise to internal distributed representations that are high-level abstractions of the input, bringing better generalization.

We study adding aggregate operators, such as summing up elements of a column of a relation, to logics with counting mechanisms. The primary motivation comes from database applications, where aggregate operators are present in all real life query languages. Unlike other features of query languages, aggregates are not adequately captured by the existing logical formalisms. Consequently, all previous approaches to analyzing the expressive power of aggregation were only capable of producing partial results, depending on the allowed class of aggregate and arithmetic operations. We consider a powerful counting logic, and extend it with the set of all aggregate operators. We show that the resulting logic satis es analogs of Hanf's and Gaifman's theorems, meaning that it can only express local properties. We consider a database query language that expresses all the standard aggregates found in commercial query languages, and show how it can be translated into the aggregate logic, thereby pro...

Deep Belief Networks (DBN) are generative neural network models with many layers of hidden explanatory factors, recently introduced by Hinton et al., along with a greedy layer-wise unsupervised learning algorithm. The building block of a DBN is a probabilistic model called a Restricted Boltzmann Machine (RBM), used to represent one layer of the model. Restricted Boltzmann Machines are interesting because inference is easy in them, and because they have been successfully used as building blocks for training deeper models. We first prove that adding hidden units yields strictly improved modeling power, while a second theorem shows that RBMs are universal approximators of discrete distributions. We then study the question of whether DBNs with more layers are strictly more powerful in terms of representational power. This suggests a new and less greedy criterion for training RBMs within DBNs. 1

This paper studies expressivity bounds for extensions of first-order logic with counting and unary quantifiers in the presence of relations of large degree. There are several motivations for this work. First, it is known that first-order logic with counting quantifiers captures uniform TC 0 over ordered structures. Thus, proving expressivity bounds for first-order with counting can be seen as an attempt to show TC 0 $ DLOG using techniques of descriptive complexity. Second, the presence of auxiliary built-in relations (e.g., order, successor) is known to make a big impact on expressivity results in finite-model theory and database theory (where logics with counting and unary quantifiers have recently been used to model query languages with aggregation). For those logics, our goal is to extend techniques from "pure" setting to that of auxiliary relations. Until now, all known results on the limitations of expressive power of the counting and unary quantifier extensions of first-order...

We study the robustness of complete sets for various complexity classes. A complete set A is robust if for any f(n)-dense set S ∈ P, A − S is still complete, where f(n) ranges from log(n), polynomial, to subexponential. We show that robustness can be used to separate complexity classes: • for every ≤ p m-complete set A for EXP and any subexponential dense sets S ∈ P, A − S is still Turing complete and under a reasonable hardness assumption even ≤ p m-complete. • For EXP and the delta levels of the exponential hierarchy we show that for every Turing complete set A and any log-dense set S ∈ P, A − S is still Turing complete. • There exists a 3-truth-table complete set A for EEXPSPACE, and a log-dense set S ∈ P such that A − S is not Turing complete. This implies that settling this issue for EEXP will either separate P from PSPACE or PH from EXP. • We show that the robustness results for EXP and the delta levels of the exponential hierarchy do not relativize. 1

We study the expressive power of counting logics in the presence of auxiliary relations such as orders and preorders. The simplest such logic, first-order with counting, captures the complexity class TC 0 over ordered structures. We also consider first-order logic with arbitrary unary quantifiers, and infinitary extensions. We start by giving a simple direct proof that first-order with counting, in the presence of preorders that are almost-everywhere linear orders, cannot express the transitive closure of a binary relation. The proof is based on locality of formulae. We then show that the technique cannot be extended to linear orders, and that the result does not say anything about the power of invariant queries in first-order with counting, in the presence of those preorders, vs. the class TC 0 . In the second part of the paper we then prove a separation result showing that for all the counting logics above, a linear order is more powerful than a preorder that is a linea...

Abstract not found

An important open question in complexity theory is whether the circuit complexity class TC 0 is (strictly) weaker than LOGSPACE. This paper considers this question from the viewpoint of descriptive complexity theory. TC 0 can be characterized as the class of queries expressible by the logic FOC(<;+;), which is first-order logic augmented by counting quantifiers on ordered structures that have addition and multiplication predicates. We show that in first-order logic with counting quantifiers and only an addition predicate it is not possible to express "deterministic transitive closure" on ordered structures. As this is a LOGSPACE-complete problem, this logic therefore fails to capture LOGSPACE. It also directly follows from our proof that in the presence of counting quantifiers, multiplication cannot be expressed in terms of addition and ordering alone. 1. Introduction The interest in finite model theory from a complexity theory point of view is motivated by the fact that "descrip...

this article, we now know that such problems are all isomorphic to the standard complete set for the complexity class, under depth-three AC

Razborov and Rudich identified an imposing barrier that stands in the way of progress toward the goal of proving superpolynomial lower bounds on circuit size. Their work on “natural proofs” applies to a large class of arguments that have been used in complexity theory, and shows that no such argument can prove that a problem requires circuits of superpolynomial size, even for some very restricted classes of circuits (under reasonable cryptographic assumptions). This barrier is so daunting, that some researchers have decided to focus their attentions elsewhere. Yet the goal of proving circuit lower bounds is of such importance, that some in the community have proposed concrete strategies for surmounting the obstacle. This lecture will discuss some of these strategies, and will dwell at length on a recent approach proposed by Michal Koucky and the author.