Neural Network

[nickoppen]

[censix]

[nickoppen]

Thanks for that censix. There's been a huge amount of work done in this area since I left uni many years ago. Do you know of any papers that compare the performance of deep networks to more shallow ones? I'm interested to see what all of that extra work buys you at the end of the day.

As for the implementation, I'm going to start with a simple, one hidden layer implementation to start with. There are a number of key areas that I'd like to know more about before attempting anything more ambitious. For example, with larger networks I think that local memory constraints might be a problem. I'm also uncertain if my whole architecture is the best one for the job.

nick

[roberto]

[censix]

[roberto]

[timpart]

[roberto]

[censix]

@Roberto

you calculation is very interesting. It shows that through the use of massive parallellization, a large 1-layer MLP may be even slightly faster to train than a much smaller 3-layer MLP.

But that does not address the core question that I still have and will continue to have until I find a good paper or a good explanation for it. And that is: Is there formal proof that a 3-layer MLP 10-A-B-C-1 is absolutely equivalent with a 1-layer 10-X-1 MLP, where X ~ A*B*C or X being some other function of A,B,C.

Until this is not clear, I would not dare to just replace N-layer MLPs by large 1-layer MLPs and just hope that it will work.

But the point about being able to parallelize(=speed up) 1-layer MLP training more efficiently than N-layer training makes sense!

[roberto]