The entropy rate of Linear Additive Markov Processes

Simple item page
dc.contributor.authorSmart, B.
dc.contributor.authorRoughan, M.
dc.contributor.authorMitchell, L.
dc.contributor.editorKovtun, V.
dc.date.issued2024
dc.description.abstractThis work derives a theoretical value for the entropy of a Linear Additive Markov Process (LAMP), an expressive but simple model able to generate sequences with a given autocorrelation structure. Our research establishes that the theoretical entropy rate of a LAMP model is equivalent to the theoretical entropy rate of the underlying first-order Markov Chain. The LAMP model captures complex relationships and long-range dependencies in data with similar expressibility to a higher-order Markov process. While a higher-order Markov process has a polynomial parameter space, a LAMP model is characterised only by a probability distribution and the transition matrix of an underlying first-order Markov Chain. This surprising result can be explained by the information balance between the additional structure imposed by the next state distribution of the LAMP model, and the additional randomness of each new transition. Understanding the entropy of the LAMP model provides a tool to model complex dependencies in data while retaining useful theoretical results. To emphasise the practical applications, we use the LAMP model to estimate the entropy rate of the LastFM, BrightKite, Wikispeedia and Reuters-21578 datasets. We compare estimates calculated using frequency probability estimates, a first-order Markov model and the LAMP model, also considering two approaches to ensure the transition matrix is irreducible. In most cases the LAMP entropy rates are lower than those of the alternatives, suggesting that LAMP model is better at accommodating structural dependencies in the processes, achieving a more accurate estimate of the true entropy.
dc.description.statementofresponsibilityBridget Smart, Matthew Roughan, Lewis Mitchell
dc.identifier.citationPLoS ONE, 2024; 19(4):e0295074-1-e0295074-13
dc.identifier.doi10.1371/journal.pone.0295074
dc.identifier.issn1932-6203
dc.identifier.issn1932-6203
dc.identifier.orcidRoughan, M. [0000-0002-7882-7329]
dc.identifier.orcidMitchell, L. [0000-0001-8191-1997]
dc.identifier.urihttps://hdl.handle.net/2440/142408
dc.language.isoen
dc.publisherPublic Library of Science (PLoS)
dc.relation.granthttp://purl.org/au-research/grants/arc/DP210103700
dc.rights© 2024 Smart et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
dc.source.urihttp://dx.doi.org/10.1371/journal.pone.0295074
dc.subjectEntropy; Markov models; Markov processes; Autocorrelation; Probability distribution; Ergodicity; Conditional entropy; Information entropy
dc.subject.meshLinear Models
dc.subject.meshProbability
dc.subject.meshMarkov Chains
dc.subject.meshAlgorithms
dc.subject.meshEntropy
dc.titleThe entropy rate of Linear Additive Markov Processes
dc.typeJournal article
pubs.publication-statusPublished

Files

Original bundle
Now showing 1 - 1 of 1
No Thumbnail Available
Name:
hdl_142408.pdf
Size:
1003.48 KB
Format:
Adobe Portable Document Format
Description:
Published version
Download

Collections

Research Outputs