Welcome Guest! To enable all features please Login or Register.

Notification

Icon
Error

Scanning UWS leads to crashes
Guest
#1 Posted : Friday, March 9, 2012 4:10:28 PM(UTC)
Groups:
Joined: 11/1/2005(UTC)
Posts: 278

Sorry for the delay, real work was getting in the way. I am still getting an application crash, but I don't think it's UltiDev. Background on this - I've used Ultidev 32 bit in other programs but this is the first time I've used it in a 64bit environment. My install was part of the install instructions from Snort, and the Kiwi Syslog server. Thoughts?
algorithm9i
#2 Posted : Friday, March 9, 2012 4:11:42 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Sorry forgot to log in before I replied - screen shot of the new error attached.
algorithm9i attached the following image(s):
algorithm9i attached the following image(s): solarwinds.jpg
algorithm9i
#3 Posted : Friday, March 9, 2012 4:40:26 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
I spoke too soon. After stepping through the errors there were additional messages from UltiDev
algorithm9i attached the following image(s):
algorithm9i attached the following image(s): debug3.jpg
algorithm9i attached the following image(s): debug1.jpg
algorithm9i attached the following image(s): debug1.jpg
algorithm9i attached the following image(s): debug.jpg
algorithm9i attached the following image(s): debug2.jpg
Ultidev Team
#4 Posted : Friday, March 9, 2012 4:53:55 PM(UTC)
Ultidev Team

Groups: Administration
Joined: 11/3/2005(UTC)
Posts: 2,253

Thanks: 28 times
Was thanked: 60 time(s) in 59 post(s)
Hi there!

Not a problem whatsoever - thank you for testing. This is interesting data. Would it be possible for us to get exported Event Log (an .evtx file) with the exceptions? If you could attach it as a password-protected ZIP file to your post, and send the password to us as Personal Message (PM), it would be very helpful.

Could you please confirm that monitoring service has restarted automatically after crash and re-started crashed host processes?

Best regards,
UltiDev Team.
Please donate at http://www.ultidev.com/products/Donate.aspx to help us improve our products.
algorithm9i
#5 Posted : Sunday, March 11, 2012 10:09:00 AM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
It does appear that the process is restarting - and the exception is being thrown by ASP.NET - Don't know why I didn't look at the application log, perhaps I was wearing the wrong hat.
Ultidev Team
#6 Posted : Sunday, March 11, 2012 5:46:25 PM(UTC)
Ultidev Team

Groups: Administration
Joined: 11/3/2005(UTC)
Posts: 2,253

Thanks: 28 times
Was thanked: 60 time(s) in 59 post(s)
Hi!

We've looked at the log and here what we have found:

  • Our design expects that application hosting process, UWS.AppHost.XXX.exe, can sometimes be crashed by an application. Although exceptions occurring within the context of serving an HTTP request normally lead to an "ASP.NET yellow page of death", in some cases exceptions may crash UWS host process as any .NET application can be crashed by an unhandled exception. For example, if an exception occurs outside of the request handling context (say you have started a thread on application_start() and that threw an exception), the host process will crash. UWS as a whole is designed to expect this, and to survive by UWS Monitoring Service sending heartbeats to host processes, and restarting the host processes if they have died.

    It is unclear from the log whether the exception that crashed UWS.AppHost process was outside of the request handling context or not:
    Quote:
    Exception: System.Runtime.Serialization.SerializationException

    Message: Unable to find assembly 'SolarWinds.KiwiSyslog.WebAccess.Core, Version=1.3.0.0, Culture=neutral, PublicKeyToken=null'.

    StackTrace: at System.Runtime.Serialization.Formatters.Binary.BinaryAssemblyInfo.GetAssembly()
    at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetType(BinaryAssemblyInfo assemblyInfo, String name)
    at System.Runtime.Serialization.Formatters.Binary.ObjectMap..ctor(String objectName, String[] memberNames, BinaryTypeEnum[] binaryTypeEnumA, Object[] typeInformationA, Int32[] memberAssemIds, ObjectReader objectReader, Int32 objectId, BinaryAssemblyInfo assemblyInfo, SizedArray assemIdToAssemblyTable)
    at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.ReadObjectWithMapTyped(BinaryObjectWithMapTyped record)
    at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
    at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
    at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
    at System.Runtime.Remoting.Channels.CrossAppDomainSerializer.DeserializeObject(MemoryStream stm)
    at System.AppDomain.Deserialize(Byte[] blob)
    at System.AppDomain.UnmarshalObject(Byte[] blob)

    If you could shed a light on what is this deserialization call is called by, we could probably see whether this is by design or there is a problem to be fixed.

  • What's more disturbing, is the other exception - the crash of UWS Monitoring Service (your screenshot is attached). Although it gets automatically restarted, that's a fail-safe, and is not supposed to happen. This is more or less guaranteed to be our bug to be fixed. Unfortunately, the log file is huge, and for whatever reason filtering and searching it is not working, so we could not find the relevant error entry in the EVTX file.

    Would it be possible for you to clear the event log on that box, then reproduce the problem, and then send us the log only with relevant items? Or if clearing the log is not an option, reproduce the problem and post the exception log text for UWS Monitoring Service crash?


Best regards,
UltiDev Team.
Ultidev Team attached the following image(s):
Ultidev Team attached the following image(s): UWS Monitoring Service crash.jpg
Please donate at http://www.ultidev.com/products/Donate.aspx to help us improve our products.
algorithm9i
#7 Posted : Sunday, March 11, 2012 6:14:30 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Hopefully this one is more managable, sorry about that guys. This is my first time working this type of issue, I'm doing my best.

PS - same password as before
1 user thanked algorithm9i for this useful post.
Ultidev Team on 3/11/2012(UTC)
Ultidev Team
#8 Posted : Sunday, March 11, 2012 6:40:58 PM(UTC)
Ultidev Team

Groups: Administration
Joined: 11/3/2005(UTC)
Posts: 2,253

Thanks: 28 times
Was thanked: 60 time(s) in 59 post(s)
No problem at all - thank you for providing very useful information!

The second log shows only the UWS AppHost crash, but no UWS Monitoring Service crash - the one that can't load "SolarWinds.KiwiSyslog.WebAccess.Core" assembly. BTW, is there an application registered that uses Kiwi Syslog? UWS internals don't use it. If you let us know about what the application that uses Kiwi does, especially whether Kiwi is used in the course of UWS serving an HTTP request, we'll be happy to assist in determining whether this is an expected behavior or our bug.

If you see that other crash, where UWS Monitoring Service has died, we'll be very thankful for the call stack trace copy, if it's recorded in the Event Log. That is most certainly something we'd like to take care of.

Thank you again!

Best regards,
UltiDev Team.
Please donate at http://www.ultidev.com/products/Donate.aspx to help us improve our products.
algorithm9i
#9 Posted : Sunday, March 11, 2012 7:00:27 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
The KiwiSyslog server I installed as part of a snort IDS/IPS installation, http://www.snort.org/ass....8.5.2_on_Windows_7.pdf

I may not have waited long enough for the scan to complete. I'll clear everything and run again on both 32 & 64 bit. Will upload as soon as it's finished.
algorithm9i
#10 Posted : Sunday, March 11, 2012 7:30:30 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
No errors at all on the 32bit install. Attached is the 3rd run on the 64 bit machine. Let me know if there's anything else you need.
Ultidev Team
#11 Posted : Monday, March 12, 2012 9:53:37 AM(UTC)
Ultidev Team

Groups: Administration
Joined: 11/3/2005(UTC)
Posts: 2,253

Thanks: 28 times
Was thanked: 60 time(s) in 59 post(s)
Hi there!

We can see only the same issue - with the failed attempt to load SolarWinds.KiwiSyslog.WebAccess.Core assembly by one of the web applications registered with UWS.

Could you please attach the "C:\ProgramData\UltiDev\WebServer\UWS.Configuration.xml" file from that machine?

We think that whatever the cause of the failure, it's not a security vulnerability. Something is wrong with some application's configuration, but we think at this point that this application is failing on its own, without the help of the scan.

Best regards,
UltiDev Team.
Please donate at http://www.ultidev.com/products/Donate.aspx to help us improve our products.
algorithm9i
#12 Posted : Monday, March 12, 2012 3:02:55 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Attached - renamed to .txt
File Attachment(s):
UWS.Configuration.txt (4kb) downloaded 202 time(s).
Ultidev Team
#13 Posted : Monday, March 12, 2012 3:32:44 PM(UTC)
Ultidev Team

Groups: Administration
Joined: 11/3/2005(UTC)
Posts: 2,253

Thanks: 28 times
Was thanked: 60 time(s) in 59 post(s)
Thank you much.

Looking at the UWS configuration files tells us that there may have something been done to machine.config on the system in question, because Kiwi assembly gets force-loaded into UWS host process even though it is not needed by any of UWS components (an there are no other applications are registered with UWS).

We think the crash induced by a force-fed third-party assembly should NOT be of a security concern.

One final test: do you get a valid page (UWS diag page) when browse to http://localhost:7756/ on that system?

All the best,
UltiDev Team.
Please donate at http://www.ultidev.com/products/Donate.aspx to help us improve our products.
1 user thanked Ultidev Team for this useful post.
algorithm9i on 3/12/2012(UTC)
algorithm9i
#14 Posted : Monday, March 12, 2012 3:37:11 PM(UTC)
Groups: Member
Joined: 3/7/2012(UTC)
Posts: 10

Thanks: 1 times
Was thanked: 1 time(s) in 1 post(s)
Yes, the UWS diag page comes up just fine. Looks like I need to go bother the SNORT.Org guys about the install instructions or something then. Thank you all for your time and interest in this issue.
Ultidev Team
#15 Posted : Monday, March 12, 2012 3:59:48 PM(UTC)
Ultidev Team

Groups: Administration
Joined: 11/3/2005(UTC)
Posts: 2,253

Thanks: 28 times
Was thanked: 60 time(s) in 59 post(s)
Thank you for bringing this to our attention. We are always happy to do thing to ensure or improve quality of our products.

FYI, we edited our previous post to state that the crash is NOT a security concern, but Kiwi does seem to be installed in a way that affects other applications. Everything depends on whether that was intentional.

Best regards,
UltiDev Team.
Please donate at http://www.ultidev.com/products/Donate.aspx to help us improve our products.
Rss Feed  Atom Feed
Users browsing this topic
Guest (2)
Forum Jump  
You cannot post new topics in this forum.
You cannot reply to topics in this forum.
You cannot delete your posts in this forum.
You cannot edit your posts in this forum.
You cannot create polls in this forum.
You can vote in polls in this forum.