How can I create an installer with a Modern UI

From NSIS Wiki

See the Modern UI Readme in the Docs\Modern UI folder.

Modern UI examples can be found in the Examples\Modern UI folder.