Cao, M.Wang, H.Freise, A.Brown, D.Zhu, Z.H.2025-10-132025-10-132025Photonics, 2025; 12(7):670-1-670-152304-67322304-6732https://hdl.handle.net/2440/147739Near-unstable cavities hold promise for reducing thermal noise in next-generation gravitational wave detectors and for enhancing light–matter interactions in quantum electrodynamics. However, operating close to the edge of geometrical stability presents significant challenges, including increased coupling to higher-order modes and heightened sensitivity to small cavity length changes and mirror imperfections. This study employs FINESSE v3 simulations to systematically investigate the modal behavior of a plano-concave cavity as it approaches instability, incorporating measured mirror surface defects and anisotropic curvature to replicate realistic conditions. The simulations highlight the degradation of beam purity and control signals as the cavity approaches instability. By validating the simulations against experimental data, we confirm FINESSE’s reliability for modeling cavities while identifying critical limitations in regimes close to the edge of stability. These findings provide essential guidance for optimizing cavity designs in future gravitational wave detectors, balancing performance gains against the challenges of operating at the stability edge.en© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/ licenses/by/4.0/).near-unstable cavity; marginally stable cavity; cavity resonators; mirror map; laser interferometry; gravitational wave detectorJournal article10.3390/photonics12070670746385Brown, D. [0000-0001-7851-3939]