Results 1  10
of
173,862
Entropy Estimation
, 1996
"... We consider two algorithm for online prediction based on a linear model. The algorithms are the wellknown gradient descent (GD) algorithm and a new algorithm, which we call EG \Sigma . They both maintain a weight vector using simple updates. For the GD algorithm, the update is based on subtr ..."
Abstract
 Add to MetaCart
We consider two algorithm for online prediction based on a linear model. The algorithms are the wellknown gradient descent (GD) algorithm and a new algorithm, which we call EG \Sigma . They both maintain a weight vector using simple updates. For the GD algorithm, the update is based on subtracting the gradient of the squared error made on a prediction. The EG \Sigma algorithm uses the components of the gradient in the exponents of factors that are used in updating the weight vector multiplicatively. We present worstcase loss bounds for EG \Sigma and compare them to previously known bounds for the GD algorithm. The bounds suggest that the losses of the algorithms are in general incomparable, but EG \Sigma has a much smaller loss if only few components of the input are relevant for the predictions. We have performed experiments, which show that our worstcase upper bounds are quite tight already on simple artificial data. 1 Introduction We consider a scenario in w...
Bias analysis in entropy estimation
 J. Phys. A Math. Gen
, 2004
"... Abstract: We consider the problem of finite sample corrections for entropy estimation. New estimates of the Shannon entropy are proposed and their systematic error (the bias) is computed analytically. We find that our results cover correction formulas of current entropy estimates recently discussed ..."
Abstract

Cited by 17 (0 self)
 Add to MetaCart
Abstract: We consider the problem of finite sample corrections for entropy estimation. New estimates of the Shannon entropy are proposed and their systematic error (the bias) is computed analytically. We find that our results cover correction formulas of current entropy estimates recently discussed
Convergence of best entropy estimates
 SIAM J. Optim
, 1991
"... Abstract. Given a finite number of moments of an unknown density on a finite measure space, the best entropy estimatethat nonnegative density x with the given moments which minimizes the BoltzmannShannon entropy I(x): = x log xis considered. A direct proof is given that I has the Kadec property ..."
Abstract

Cited by 40 (7 self)
 Add to MetaCart
Abstract. Given a finite number of moments of an unknown density on a finite measure space, the best entropy estimatethat nonnegative density x with the given moments which minimizes the BoltzmannShannon entropy I(x): = x log xis considered. A direct proof is given that I has the Kadec
Multicollinearity and Maximum Entropy Estimators
 Economics Bulletin
, 2001
"... Multicollinearity hampers empirical econometrics. The remedies proposed to date suffer from pitfalls of their own. The ridge estimator is not generally accepted as a vital alternative to the ordinary least−squares (OLS) estimator because it depends upon unknown parameters. The generalized maximum en ..."
Abstract

Cited by 5 (0 self)
 Add to MetaCart
entropy estimator depends upon subjective exogenous information. This paper presents a novel maximum entropy estimator that does not depend upon any additional information. Monte Carlo experiments show that it is not affected by any level of multicollinearity and dominates the OLS estimator uniformely
Undersmoothed kernel entropy estimators
 IEEE Transactions on Information Theory
"... Abstract—We develop a “plugin ” kernel estimator for the differential entropy that is consistent even if the kernel width tends to zero as quickly as 1=N, where N is the number of independent and identically distributed (i.i.d.) samples. Thus, accurate density estimates are not required for accurat ..."
Abstract

Cited by 4 (0 self)
 Add to MetaCart
Abstract—We develop a “plugin ” kernel estimator for the differential entropy that is consistent even if the kernel width tends to zero as quickly as 1=N, where N is the number of independent and identically distributed (i.i.d.) samples. Thus, accurate density estimates are not required
Coverage Adjusted Entropy Estimation
 TO APPEAR IN A SPECIAL ISSUE OF STATISTICS IN MEDICINE ON NEURONAL DATA ANALYSIS
, 2007
"... Data on “neural coding” have frequently been analyzed using informationtheoretic measures. These formulations involve the fundamental, and generally difficult statistical problem of estimating entropy. We review briefly several methods that have been advanced to estimate entropy, and highlight a m ..."
Abstract

Cited by 2 (2 self)
 Add to MetaCart
Data on “neural coding” have frequently been analyzed using informationtheoretic measures. These formulations involve the fundamental, and generally difficult statistical problem of estimating entropy. We review briefly several methods that have been advanced to estimate entropy, and highlight a
Universal erasure entropy estimation
 In Proc. of the 2006 IEEE Intl. Symp. on Inform. Theory, (ISIT’06
, 2006
"... Abstract — Erasure entropy rate (introduced recently by Verdú and Weissman) differs from Shannon’s entropy rate in that the conditioning occurs with respect to both the past and the future, as opposed to only the past (or the future). In this paper, universal algorithms for estimating erasure entrop ..."
Abstract

Cited by 7 (3 self)
 Add to MetaCart
Abstract — Erasure entropy rate (introduced recently by Verdú and Weissman) differs from Shannon’s entropy rate in that the conditioning occurs with respect to both the past and the future, as opposed to only the past (or the future). In this paper, universal algorithms for estimating erasure
Fast kernel entropy estimation and optimization
 Signal Processing
, 2005
"... Differential entropy is a quantity used in many signal processing problems. Often we need to calculate not only the entropy itself,but also its gradient with respect to various variables,for efficient optimization,sensitivity analysis,etc. Entropy estimation can be based on an estimate of the probab ..."
Abstract

Cited by 9 (4 self)
 Add to MetaCart
Differential entropy is a quantity used in many signal processing problems. Often we need to calculate not only the entropy itself,but also its gradient with respect to various variables,for efficient optimization,sensitivity analysis,etc. Entropy estimation can be based on an estimate
A MaximumEntropyInspired Parser
, 1999
"... We present a new parser for parsing down to Penn treebank style parse trees that achieves 90.1% average precision/recall for sentences of length 40 and less, and 89.5% for sentences of length 100 and less when trained and tested on the previously established [5,9,10,15,17] "stan dard" se ..."
Abstract

Cited by 963 (19 self)
 Add to MetaCart
" sections of the Wall Street Journal tree bank. This represents a 13% decrease in error rate over the best singleparser results on this corpus [9]. The major technical innova tion is the use of a "maximumentropyinspired" model for conditioning and smoothing that let us successfully to test
Results 1  10
of
173,862