rdipietro / jupyter-notebooks Goto Github PK

The code uses only one training example.
What should I add to perform a mini-batch training?
Coud I use tensorflow.scan over batch_size?

there is an error in the first equation

Hi,

Great article!
I have a question, not sure if it's OK if I post this here as an issue. If it's not I'll delete it.

When talking about encoding elements of a distribution in order to minimise the number of bits, i your article says this:

It turns out that if you have access to the underlying distribution y, then to use the smallest number of bits on average, you should assign log(1/y_i) bits to the i-th symbol.

Why log(1/y_i)?

How can we prove that this is the minimum?

Hi Rob!

I enjoy reading "A Friendly Introduction to Croo-Entropy Loss", but I have a few questions about the notation used, because I don't think that I've come across of it before.

I think the notation is explained well enough in the post, but I'm wondering if it's a standard notation in some fields, and if it is, if you could point me towards a webpage/pdf providing an overview of the notation :)

It's the red parts which is new to me.

Hi, I am a beginner in TensorFlow and I run into your brilliant scan tutorial. I wonder whether you could share some knowledge of iterating across multiple tensors at once and/or returning tuples from fn.

So far I've tried passing a list of tensors (possibly wrapped by tf.identity) but it doesn't work.

I believe this would be valuable extension of your post :)

Petr

I am not sure if everyone can reproduce this issue, but in the Content section for Cross Entropy and About Me, I am not able to open the links
Maybe you can give links like:

Contents

...

* [Cross Entropy](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/#Cross-Entropy)
* [About Me](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/#About-Me)

...

and it should work like a charm, something like:

Contents

...

• Cross Entropy
• About Me

...

Actually I believe that you can try giving hyperlinks in markdown using the same structure, i.e, instead of using <url>/cross-entropy, use <url>/Cross-Entropy (usage of same letters as in text, and separating them with hyphens) and it should work for all links in Contents Section

First, thanks for the excellent tutorial! I've always wondered about the relationship between entropy and cross-entropy loss, and you explained it perfectly.

I did notice a possible typo in the "Predictive Power" section:

is just the first entry of y^(1)=(0.4,0.1,0.5)T , which is y1(1)=0.4 .

Shouldn't that last y1(1) be y^1(1)?

I am miles off an understanding here i assume and this is my first time raising an issue on git so please i beg everyone's indulgence
The post makes the point that

Because we usually assume that our samples are independent and identically distributed, the likelihood over all of our examples decomposes into a product over the likelihoods of individual examples:
L({y(n)},{y^(n)})=∏nL(y(n),y^(n))

(sorry first issue on git, do not know how to math)
Here is what my issue is:

1. How come the samples are identically distributed? Do we not have actual probability distributions of each sample as 0 for each value except a 1 at the actual label place. For example (from the post):

To keep going with this example, let's assume we have a total of four training images, with labels {landscape, something else, landscape, house}, giving us ground-truth distributions y(1)=(1.0,0.0,0.0)Ty(1)=(1.0,0.0,0.0)T , y(2)=(0.0,0.0,1.0)Ty(2)=(0.0,0.0,1.0)T , y(3)=(1.0,0.0,0.0)Ty(3)=(1.0,0.0,0.0)T , and y(4)=(0.0,1.0,0.0)Ty(4)=(0.0,1.0,0.0)T

so are not y(1), y(2), y(3), y(4) having different distributions?

2. Is anywhere the identically distributed part needed. Is not the entire mathematics in post consistent with just the independent distributions assumption?

I can never remember whether y_hat is supposed to be the target value or the prediction output, which makes skim reading your article quite difficult.

Two possible solutions...

1. Add a position:fixed legend that stays visible.
2. Use clearer names, e.g. p for the prediction output, t for the true value.