In order to debug a dialog screen, you need to create an SAP shortcut with the properties shown below. This can then be dragged onto the dialog window which will activate debugging mode.