Exponential families and natural parameters

[botev]

It would really great if there is a way of constructing an easy way of switching between standard and natural parameters. This particularly is to address more advanced techniques such as Structured VAEs - https://arxiv.org/abs/1603.06277. However, this would require significant thought on how to incorporate this in the API and so I think a discussion here would be good to make. I have not seen so far a good abstraction for this in any of the existing probabilistic frameworks in the community.

[thjashin]

Great idea. Exponential families are always linked to conjugacy, which is very hard to have an automated design. I guess that's why existing frameworks don't have a good support (though some claim to ). But we may start from the easy case. Maybe first a general distribution named ExponentialFamily? And let all subclasses have the natural parameter attributes.

[botev]

Just for some pointers, do you know any frameworks which have/claim to have some support of this as it would be good to see what people have come up in terms of ideas and what problems they got in the way?

[thjashin]

Edward has some preliminary design of conjugacy. But they are doing this based on graph copying, which is a feature that TF people just don't like (see this PR by me). This design seems to be deprecated by the author of Edward. They are currently working on a new version based on function reuse like us, but it's not clear what they will do about the original conjugacy features.

[thjashin]

I'm also interested in implementing some examples shown in the structured vae paper. That's a very neat paper. If you could outline some detailed needs, I'm happy to help with it.

[thjashin]

Update: @liyr will start working on a general abstraction for exponential families.

[botev]

I had some thoughts on this and I think essentially what is needed is something similar to the query, but which given some observations if the structure of the (remaining) graph is conjugate gives you back the parameters (natural or not) of the exact posterior or throws an error if it is not conjugate. The main challenges is basically how to tag things such that you can indeed detect if things are conjugate or not.

[thjashin]

I like it. That's the ultimate goal. We are working at the very beginning though, to have some an ExponentialFamily abstraction, and something like Normal.from_natural_parameter(). Detecting conjugacy for single-particle models is easy, But it's getting hard as you have many plates in the graphical model.

[sameerkhurana10]

Hi,

Any updates on this?

[thjashin]

We have some attempts here, but they are far from perfect. @sameerkhurana10