My initial thought is to start some dummy EXE that closes itself after 1 second, then use that to test. Is that out of the realm of Unit Testing?

Is this a good test? Sure, so create it. Is it “a Unit Test” in the real sense of the word? I don’t think so, I would call this a “system test” or something like that, but that does not make the test less valuable.

Mock it out at a higher level than that. Create a proxy class around Process.Start(), fake that out in the test and check the input.

public interface IProcessProxy
{
     ProcessInfo Start(string application, string[] arguments);
}

public class ProcessProxy : IProcessProxy
{
    public ProcessInfo Start(string application, string[] arguments)
    {
        return Process.Start(application, arguments);
    }
}

// You could use a mocking framework for this, but for the purposes
// of this example ...
public class FakeProcessProxy : IProcessProxy
{
    private string _expectedApplication;
    private string[] _expectedArguments;
    private ProcessInfo response;

    public FakeProxy(string expectedApplication, string[] expectedArguments, ProcessInfo response)
    {
         _expectedApplication = expectedApplication;
         _expectedArguments = expectedArguments;
    }

    public ProcessInfo Start(string application, string[] arguments)
    {
         // compare input to expectations and throw exception if not matching
         return _response;
    }
}

// You can also use an IoC framework to inject your IProcessProxy, but I won't.
public class ClassUnderTest
{
    public ClassUnderTest(IProcessProxy proxy)
    {
        _proxy = proxy;
    }

    public ClassUnderTest() : this(new ProcessProxy())
    {
    }

    public void MethodUnderTest()
    {
        // Do stuff

        ProcessInfo process = _proxy.Start(@"C:Program FilesAppApp.exe", new[] { "arg1", "arg2" });
        process.WaitForExit();

        if (process.ExitCode == 0)
        {
            // Act on success
        }
        else
        {
            // Act on failure
        }
    }   
}

Wherever you need to consume ClassUnderTest in application code, use the default constructor. In your tests, pass a FakeProcessProxy to the other constructor, using your expected Proxy Start parameters and your test result in the fake’s constructor.

When strictly following the philosophy of unittesting (emphasis on unit), you should not create an Exe-file, but instead test if your class calls the interfaces to spawning and monitoring that process correctly.
After all, you want to test your class, not the library responsible for process handling.

But from a pragmatic point of view, your approach is fine, though 1 second seems to be a little long.

I did something similar, but just called ping localhost. Saves the hassle of putting executables on your build server