For more information on this, see the issue on GitHub amphtml#22714. To test, you must have the code from amphtml#24917 applied to your AMP local dev environment and then served via the AMP Local Proxy browser extension.
The following examples were taken from the Pym.js. By the way, the large blue header above is there to help avoid situations where an iframe is in the initial viewport, since in that case the overflow button will need to be shown to prevent content shifting. Learn more about iframe resizing in AMP. After loading the page and testing the iframes, scroll to the bottom and then reload the page so that the iframes are then in the initial viewport, and then you should see the overflow button. This is working as intended.