Loading BuildingSceneLayers from a portal item loads building filters if there are any stored on the portal data. However, loading BuildingsSceneLayers from a Scene Service URL does not automatically load a BuildingFilter and this must be set up.
Refer to the following example on how to create the BuildingFilter for scene services: https://codepen.io/SaschaBrunnerCH/pen/abwjZJZ?editors=1000