Information Theory notes #01

Short notes with a little bit of clarification.

Self information of the event is proportional to probability of event occurrence. The more likely event is the less information it holds.

It could be expressed in the following formula.

$$I(x) = - logP(x)$$
Given that $ D(x) \in [0,1] $

From graph it's clear that event's with less probability are more informative. Ideally guaranteed event shouldn't provide information. $ P(x)=1 $
More details about this topic could be found in  Yoshua Bengio's book chapter  Probability and Information Theory
http://www-labs.iro.umontreal.ca/~bengioy/dlbook/

bit tricks

1. check if number is power of 2
The trick is that, if number is a power of two it contains only single bit with value 1.
x & (x-1) unsets right most bit with value 1. And in this case it unsets the only bit with value 1, if number is a power of 2.

idea was taken from this article:
http://articles.leetcode.com/2010/09/number-of-1-bits.html

check_base_2 = lambda x: x and not(x & (x-1))

#bin(16)   #10000#bin(16-1) #01111
#  10000# &#  01111#  00000
assert check_base_2(16)
assert check_base_2(32)
assert check_base_2(128)
assert check_base_2(8)
assert check_base_2(256)
assert check_base_2(3) != True