In the post processor text field you can use the $EXECUTABLE variable to reference the executable that is currently being post-processed. The working directory of the executed process is the directory your config is located in so you can use relative filenames for key or certificate files. If the signing command cannot replace the executable directly, but rather needs a separate output file, use the $OUTFILE variable. It will receive a temporary output file name that will be moved back to the processed executable by install4j after the post processor has completed.
The tool has the same syntax as the one from Microsoft. A typical entry would be mono /opt/install4j/resource/signcode.exe -spc mycert.spc -v mykey.pvk -vp password -t http://timestamp.verisign.com/scripts/timstamp.dll $EXECUTABLE
Some SPC files cannot be read directly by this tool. If this is the case for your certificate, you can export all CER files from the SPC file and generate a new SPC file with the cert2spc tool included with mono. You have to add the CER files in the order of the certificate chain (your own certificate is the last one on the command line).