Integrating DigiCert Click-to-Sign with InstallShield to sign both setup.exe and bundled MSI file

All apps submitted to the Windows Store must be signed with Authenticode digital signatures, and consequentially must not generate any SmartScreen warnings, regardless of whether these warnings are false positives, before the apps can be approved by Microsoft. This requirement was one of the challenges I faced during my latest Windows Store app submission. I had first tried to upload the binaries without signing them at all, only for the upload to fail the automatic validation process due to the lack of digital signature. I then tried to sign the file locally using a self-signed cert using signtool:

signtool sign /a /fd SHA256  /tr http://timestamp.digicert.com /td SHA256 MyFile.exe

signtool.exe is part of the Windows Kits, which is usually installed as part of the Windows SDK during Visual Studio installation. On my computer, signtool can be located at C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64. For more information about signtool, read this article by Microsoft.

With the code signing signature added, my setup.exe passed the automatic validation process and was submitted successfully for review. No SmartScreen warnings were generated when the setup.exe was run on my computer, or on a fresh Windows 10 installation. There was still a User Account Control (UAC) warning asking if user wants to installed an app created by an unknown publisher, which I thought was normal. However, a week later, Microsoft rejected the binary, stating that SmartScreen warnings were still generated, without providing further details. The rejection message also stated that I could request Microsoft to whitelist the file by downloading the file using Microsoft Edge, choose to report the file as safe when presented with the SmartScreen warnings, fill in the form, waited 24 hours and submit the app again. I tried those steps multiple times only for the app to be rejected again with the same SmartScreen issues.

At this point I decided to dig deeper into the issue and found out that the executables must be signed with a known Certificate Authority (CA), such as DigiCert, and not just a self-signed certificate. The publisher name would also be validated and the setup.exe installer would no longer generate unknown publisher warning messages when executed. Following the advice, I then purchased and set up my DigiCert Authenticode account, which took almost two days due to the complexity of the steps. Next, I installed Digicert Click-to-Sign by first downloading Keylockertools-windows-x64.msi from Digicert portal and install it. I then opened C:\Program Files\DigiCert\DigiCert Keylocker Tools and within that folder, run DigiCert_Click_to_sign.msi. With this, I was finally able to right click my setup.exe file and choose DigiCert Click-to-Sign > Sign Now:

If everything is setup correctly, you will receive a message saying that signing has been completed and the properties panel for the executable should contain a new tab named Digital Signatures with the correct publisher name and signature date. If you receive a message indicating signing has failed, on the same menu, choose Settings and review the signing credentials. If you still cannot get things to work, open C:\Program Files\DigiCert\Click-to-sign\log4net.config and find out the location of the signing log file. On my installation, the path is at C:\Users\[username]\.signingmanager\logs. If the log complained signtool cannot be found, edit system’s PATH variable and add an entry to the full path of signtool on your machine. If the log refers to old signing keys or paths, reboot your computer and try again, as Click-to-Sign seems to cache these settings in memory.

Once setup.exe had been signed, I noticed the unknown publisher message disappeared and was confident that the app would be approved for the Windows Store. Three days later, the rejection came, still complaining about SmartScreen issues. This time however, the rejection indicated that the SmartScreen warnings were found during uninstallation, and not during initial setup. After further research, it turned out that because my installer was packed using InstallShield, setup.exe would only extract the MSI file and it was this MSI file which handled both the installation and uninstallation of the apps. Once the app had been installed, Windows will cache the MSI file in case the app is uninstalled later. Because the MSI file is not signed, SmartScreen warnings would then be generated during uninstallation.

To fix this, the InstallShield Visual Studio project will need to be modified so that both the MSI and setup.exe files will be signed. To achieve this, the first course of action is to find the command used by DigiCert to sign the executable. Luckily this command can easily be found in the log file for Click-to-Sign, illustrated below:

C:\Program Files\DigiCert\DigiCert Keylocker Tools>smctl sign --keypair-alias=key_847828698 -d=SHA256 --verbose --config-file "Select the pkcs11 configuration file" --input "C:\temp\file.msi"

Command :
 signtool sign  /tr http://timestamp.digicert.com /td SHA256  /fd  SHA256   /csp "DigiCert Signing Manager KSP" /kc "key_XXXXX" /f "C:\Users\username\AppData\Local\Temp\2806843057\key_XXXXX_certificate.pem"  "C:\temp\file.msi"
Output :
 Done Adding Additional Store
Successfully signed: C:\temp\file.msi

Although DigiCert Click-to-Sign settings contain an option for pkcs11, I have found it to be optional and can be left blank. Leaving this setting empty will however result in “Select the pkcs11 configuration file” being passed as a command line parameter to smctl, which is normal. key_XXXXX is the name of the key that has been created for local signing during Click-to-Sign setup. The value of XXXXX will not change until you update your Click-to-Sign settings with new credentials.

The next task is to find out how to configure InstallShield. Although InstallShield contains some settings for signing configuration under Media > Releases > Express > Single Image > Signing, these simplified signing settings cannot be used with DigiCert:

The Events tab on the same screen can however be used to sign the installer with Digicert. Specifically, the Precompression Event, triggered after the MSI has been created but before setup.exe is generated, and the Postbuild Event, triggered after setup.exe (which contains the MSI) has been generated, can be used to sign both the MSI and setup.exe respectively. To do this, I wrote sign_file.bat which uses Digicert to sign a file given its path, and set this batch file as parameters for both fields:

This is the content of sign_file.bat:

@echo off
REM Check if the first parameter is provided
if "%~1"=="" (
    echo No parameters provided. Exiting...
    exit /b 1
)

REM Set the PATH to include signtool
set PATH=%PATH%;C:\Program Files (x86)\Windows Kits\10\bin\10.0.22621.0\x64

REM Check if /F is the second parameter
if "%~2"=="/F" (
    REM If /F is provided, use the entire first parameter as the input file
    set INPUT_FILE="%~1"
) else (
    REM Otherwise, construct the path using the predefined location and the filename
    set INPUT_FILE="%<ISReleasePath>%\Express\SingleImage\DiskImages\DISK1\%~1"
)

REM Call the DigiCert signing tool with the input file path
"C:\Program Files\DigiCert\DigiCert Keylocker Tools\smctl.exe" sign --keypair-alias=key_XXXXX -d=SHA256 --verbose --config-file "Select the pkcs11 configuration file" --input %INPUT_FILE%

REM Optionally, pause to keep the command prompt open
REM pause

The script also accepts a second /F parameter, which, if passed, will assume that the provided input file includes its full path. If this parameter is not passed, the input file will be assumed to be present in InstallShield’s output directory (Express\SingleImage\DiskImages\DISK1). These assumptions work for my setup but might need to be modified to match your InstallShield installation.

Before using this script, you should first build your InstallShield project a few times and observe where it creates the MSI and setup.exe files, and update the path as needed. The full command to smctl.exe, particularly the local key name (key_XXXXX) will also need to be updated to match your installation. Because InstallShield runs the script in a popup console window visible to the user, I have optionally added a PAUSE command at the end requiring the user to press a key to exit the script (and close the console window). This way, any error messages generated will be visible. Once the script has been tested and performs satisfactorily, the PAUSE command can be commented out.

On a side note, on an InstallShield Basic MSI project (the type of project created by InstallShield for Visual Studio), to install the app silently for Windows Store submission, run setup.exe /s /v”/qn”. To extract the MSI from setup.exe after it has been created, run setup.exe /s /x /b”installer” /v”/qn” which will create the MSI in a sub-folder named installer.

With both the MSI and setup.exe files signed, uninstalling my app will no longer generate any SmartScreen warnings, and the app was finally approved for Windows Store. It should be noted that the app name and publisher name as shown on the store listing must match the values created in Control Panel or the app may be rejected. Also, Windows Store will take the app version number from the setup.exe installer and display to the user, and not from your main executable. Finally, since SmartScreen warnings are triggered when launching executables and not when DLLs are loaded, only the MSI, setup.exe and the app main executable need to be code-signed. DLLs used by the apps, both managed (.NET) or unmanaged (C/C++), need not be signed. It would also be impractical and expensive to sign all involved DLLs, as a typical app can easily load multiple DLLs at the same time. Mismatching signed and unsigned DLL may also cause DLL loading issues that can be hard to debug.