CDOoDocuments.StdDocumentDescDocuments.DocumentDescContainers.ViewDescViews.ViewDescStores.StoreDescu"Documents.ModelDescContainers.ModelDescModels.ModelDescStores.ElemDesc! !TextViews.StdViewDescTextViews.ViewDesce!TextModels.StdModelDescTextModels.ModelDesc\ T TextModels.AttributesDesc'* '* 8*PS%.StdFolds.FoldDesc4 &  3* )τ)  6 *# author-topic model (Steyvers et al. 2004) model{ for( m in 1 : M ) { for( n in 1 : N ) { x[m,n] ~ dcat(a[m, 1:A]) z[m,n] ~ dcat(theta[x[m,n],1:K]) w[m,n] ~ dcat(phi[z[m,n] , 1:V]) } } for( k in 1 : K ) { phi[k , 1:V] ~ ddirch(beta[]) } for ( ii in 1 : A ) { # this was theta[ii,1:K] theta[ii,1:K] ~ ddirch(alpha[]) } } TextControllers.StdCtrlDescTextControllers.ControllerDescContainers.ControllerDescControllers.ControllerDesc TextRulers.StdRulerDescTextRulers.RulerDescTextRulers.StdStyleDescTextRulers.StyleDescZTextRulers.AttributesDesc$0ZGo *    8* .+ .   3*D% M sk6 *+# latent Dirichlet allocation (Blei et al. 2002) model{ for( m in 1 : M ) { theta[m,1:K] ~ ddirch(alpha[]) for( n in 1 : N ) { z[m,n] ~ dcat(theta[m,1:K]) w[m,n] ~ dcat(phi[z[m,n] , 1:V]) } } for( k in 1 : K ) { phi[k , 1:V] ~ ddirch(beta[]) } } aY?$0ZGo *    8* .S [.   &.    '* .*:+T2Q E kck '*|5 *# data for author-topic model (Steyvers et al. 2004) list( # words (numbers are terms) w = structure(.Data = c( # document 1 1, 4, 3, 2, 3, 1, 4, 3, 2, 3, 1, 4, 3, 2, 3, # document 2 2, 2, 4, 2, 2, 2, 2, 4, 2, 2, 2, 2, 4, 2, 2, # document 3 1, 6, 5, 6, 7, 1, 6, 5, 6, 7, 1, 6, 5, 6, 7, # document 4 5, 6, 6, 2, 6, 5, 6, 2, 2, 6, 5, 6, 6, 6, 7 ), .Dim = c(4,15)), # vocabulary size V = 7, # corpus size M = 4, # max document size N = 15, # number of concepts K = 2, # number of actors A = 3, # hyperparameters (constant) alpha = c(4,4), beta = c(.1,.1,.1,.1,.1,.1,.1), a = structure( .Data = c( # document 1 .5, .5, 0, # document 2 1, 0, 0, # document 3 0, .5, .5, # document 4 .3, .3, .4 ), .Dim = c(4,3)) )  aY?$0ZGo *   .*.: !.'   3*S4]1  k '*|9 *# data for latent Dirichlet allocation (Blei et al. 2002) list( w = structure(.Data = c( # document 1 1, 4, 3, 2, 3, 1, 4, 3, 2, 3, 1, 4, 3, 2, 3, # document 2 2, 2, 4, 2, 2, 2, 2, 4, 2, 2, 2, 2, 4, 2, 2, # document 3 1, 6, 5, 6, 7, 1, 6, 5, 6, 7, 1, 6, 5, 6, 7, # document 4 5, 6, 6, 2, 6, 5, 6, 2, 2, 6, 5, 6, 6, 6, 7 ), .Dim = c(4,15)), # vocabulary size V = 7, # corpus size M = 4, # max document size N = 15, # number of concepts K = 2, # hyperparameters (constant) alpha = c(4,4), beta = c(.1,.1,.1,.1,.1,.1,.1) )  aY?$0ZGo *  .V ,#  '*| *|Multinomial Mixture Models Gregor Heinrich 2005-03-07 This file describes two models from the family of hierarchical multinomial mixtures: the Author-Topic Model (ATM, Steyvers et al. 2004, Rosen-Zvi et al. 2004) and Latent Dirichlet Allocation (Blei et al. 2002, Griffiths 2002, Griffiths and Steyvers 2004). Models Their WinBUGS code is as follows: Open to view ATM WinBUGS code Open to view LDA WinBUGS code Brief explanation of these two hierarchical multinomial mixture models: The first model (author-topic model, ATM) reads out documents and tries to match authors, a, and words, w, to topics, z: Eventually the author-topic assignments are in the matrix theta[author,topic] and the topic-word assignments in phi[topic,word]. Both matrices have (normalised) multinomial parameters along their rows. The second model (latent Dirichlet allocation, LDA) does something similar, but for documents and words, each assigned to topics z. In the end the document-topic assignments are in theta[document,topic] and the topic-word assignments in phi[topic,word]. The difference is that in LDA a document index iterates monotonically during the sampling process while with the ATM, the index into theta[author,topic] is changed according to the author, x, of a word, i.e., randomly chosen among the observed co-authors of the specific document, a[m,:]. Data Sets For the author-topic model I have used the tiny data set below: Open to view data For reference, latent Dirichlet allocation (LDA) uses the following data: Open to view data Results The LDA model is a good reference for the general behaviour of the multinomial models. In WinBUGS it gives plausible results when one looks at the mean of the phi[,] and theta[,] elements (sure-fire: burn-in=10000, sampling=10000). With the code and data given, I've got the sampled mean values for LDA: phi = [ 0.1399 0.0104 0.008706 0.005712 0.2091 0.4858 0.1402; 0.05592 0.5564 0.1837 0.1885 0.004674 0.0053 0.005494 ] theta = [ 0.2634 0.7366 ; 0.1786 0.8214; 0.7942 0.2058; 0.697 0.303 ] The ATM results in the following values: phi = [ 0.161, 0.006, 0.037, 0.007, 0.196, 0.463, 0.129; 0.035, 0.585, 0.156, 0.194, 0.010, 0.009, 0.011; ] theta = [ [0.168, 0.832] [0.669, 0.331] [ 0.784, 0.216] ] References Blei, D., Ng, A., & Jordan, M. (2002). Latent Dirichlet allocation. Advances in Neural Information Process- ing Systems 14. Cambridge, MA: MIT Press. Griffiths, T. L., & Steyvers, M. (2004). Finding scien- tific topics. Proceedings of the National Academy of Sciences, 101, 5228ff. Griffiths, T. L. (2002). Gibbs sampling in the generative model of Latent Dirichlet Allocation. Working paper, Stanford Univ., www-psych.stanford.edu/~grufydd/cogsci02/lda.ps Rosen-Zvi, M., Griffiths, T., Steyvers, M., & Smyth, P. (2004). The author-topic model for authors and documents. 20th Conference on Uncertainty in Ar- tificial Intelligence. Steyvers, M., Smyth, P., Rosen-Zvi, M., & Griffiths, T. (2004). Probabilistic author-topic models for in- formation discovery. The Tenth ACM SIGKDD In- ternational Conference on Knowledge Discovery and Data Mining. aY?$0ZGo * <[ @Documents.ControllerDesc Ws8 [h