Rob,
"Running" status means that listener is ready to accept connections on application's port. If application has "Always keep in memory..." flag clear in application settings, then the application may not be even loaded despite the "Running" status. Such applications behave like ones under IIS - they get unloaded by ASP.NET if they sat idle for a while. To save memory just clear "Always keep in memory..." flag and it will be unloaded when it is serving requests for a while, even though status will still be "Running".
Also, just FYI, "Not Running" status means that application is either being loaded, or if you see it after Cassini service was running for a while, that application has failed to load for some reason.
Best regards,
UltiDev Team.
Please donate at
http://www.ultidev.com/products/Donate.aspx to help us improve our products.