Slowness when editing the page layout via the panels wizard of the administration section on large wikis

When editing the page layout via the panels wizard of the administration section, the response time becomes particularly slow on some wikis while other pages get loaded up to 100 times faster. One possible cause could be that all panels get rendered on that page, with some of them possibly requiring a large amount of time?

Possible workarounds ideas:

• Make it possible to disable the panels that are known to be potentially slow?
• Disable the panel wizard by default until this issue is fixed?