Help with keywords- Handling a orphaned object

[Banana]

A while ago, I implemented a library which gave my application extra functionality. However, because the object is created by the library rather than the application there's a chance where if the application crash or terminates improperly, the object becomes orphaned and thus binds up the system's resources.

I need to have a means of squashing that orphaned object and freeing up the resources, but since the application crashed, I no longer have the reference to the object. Therefore, what keywords should I use to research about recovering orphaned objects?

[ShadowOfGed]

A while ago, I implemented a library which gave my application extra functionality. However, because the object is created by the library rather than the application there's a chance where if the application crash or terminates improperly, the object becomes orphaned and thus binds up the system's resources.

I need to have a means of squashing that orphaned object and freeing up the resources, but since the application crashed, I no longer have the reference to the object. Therefore, what keywords should I use to research about recovering orphaned objects?

Can we get more specific details? What kind of object? What kind of resources do you need to free? If an application crashes, the kernel is going to automatically release any external handles that were open (file descriptors, sockets, shared memory, etc.) So what kind of resource are you allocating that the kernel won't clean up automatically when a process terminates?

All process terminations look the same to the kernel: a crash, a clean exit by returning from main(), a premature exit by calling exit(), or exiting on a signal like KILL, TERM, or ABRT, which used by assert(). It stops scheduling the process and cleans up any remaining resources. On a clean exit, there's not much left to clean, but that's not the point.

So… what do you need to clean up that you think the system can't? I'm at a loss right now…

[Banana]

No wonder why I was getting nowhere- obivously you can see I didn't quite have a firm understanding here.

I had assumed that the issue was the library because when application crashes, the TCP port used for SSH object is still alive, so I thought it was the library that wasn't cleaned up properly as it didn't get the required signal from the application to destroy the instance.

Did that help?

[ShadowOfGed]

No wonder why I was getting nowhere- obivously you can see I didn't quite have a firm understanding here.

I had assumed that the issue was the library because when application crashes, the TCP port used for SSH object is still alive, so I thought it was the library that wasn't cleaned up properly as it didn't get the required signal from the application to destroy the instance.

Did that help?

On which end?

The crashed client's end of the TCP connection should close automatically. The server's side of the TCP connection is liable to remain open until the connection times out.

EDIT: or is your application opening an SSH connection by spawning SSH in a *separate process*? If you're spawning a separate process to handle the SSH connection, it's possible that won't get terminated when the parent process dies.

Though, I'd expect if you're using pipes to STDIN/STDOUT on a separate SSH process that the SSH process would see a broken pipe (SIGPIPE) when the parent (your application's) process dies.

Though… you're on Windows, aren't you? If you're on Windows, half of what I say may be a lie. Windows handles sockets very differently from UNIX. In UNIX, files, local sockets and TCP sockets are all "file descriptors" to the kernel. On Windows, they're separate concepts. I have no idea what kind of cleanup Windows does upon a crash. I also have no idea how Windows handles pipes or anything like that.

REEDIT: more comments about socket closure and TCP. The kernel (on UNIX) will indeed close the file descriptors related to a socket. This terminates the client side. For UNIX domain sockets, both processes are local, and the kernel can notify the other end that a client socket closed. For TCP sockets, it won't send such a notification because there are no assumptions about the remote end of the connection. Sockets can be shutdown() before they are close()d, and I'm guessing that on a crash, the kernel may free its local data structures without notifying the remote end.

It's all about what the kernel can know about *both* ends of a connection/file/whatnot. When it knows about both ends (local files, or UNIX domain sockets) it can perform more cleanup. When it knows nothing about the remote end (TCP sockets), it can only clean up the parts it knows about. That's why TCP connections time out on the server end when a client crashes, versus closing immediately when a client exits cleanly (at least, I'd assume so).

I'm not 100% sure; this is just "informed rambling." :)

[Banana]

On which end?

The crashed client's end of the TCP connection should close automatically. The server's side of the TCP connection is liable to remain open until the connection times out.

EDIT: or is your application opening an SSH connection by spawning SSH in a *separate process*? If you're spawning a separate process to handle the SSH connection, it's possible that won't get terminated when the parent process dies.

This is on Windows, a client end. I am assuming it's a separate process because this is called using a type library, and when the application crash, I can see that TCP is still in use.

Though, I'd expect if you're using pipes to STDIN/STDOUT on a separate SSH process that the SSH process would see a broken pipe (SIGPIPE) when the parent (your application's) process dies.

I'm not using a shell. It's simply there for port forwarding, and is intended to be transparent to the users.

Though… you're on Windows, aren't you? If you're on Windows, half of what I say may be a lie. Windows handles sockets very differently from UNIX. In UNIX, files, local sockets and TCP sockets are all "file descriptors" to the kernel. On Windows, they're separate concepts. I have no idea what kind of cleanup Windows does upon a crash. I also have no idea how Windows handles pipes or anything like that.

REEDIT: more comments about socket closure and TCP. The kernel (on UNIX) will indeed close the file descriptors related to a socket. This terminates the client side. For UNIX domain sockets, both processes are local, and the kernel can notify the other end that a client socket closed. For TCP sockets, it won't send such a notification because there are no assumptions about the remote end of the connection. Sockets can be shutdown() before they are close()d, and I'm guessing that on a crash, the kernel may free its local data structures without notifying the remote end.

It's all about what the kernel can know about *both* ends of a connection/file/whatnot. When it knows about both ends (local files, or UNIX domain sockets) it can perform more cleanup. When it knows nothing about the remote end (TCP sockets), it can only clean up the parts it knows about. That's why TCP connections time out on the server end when a client crashes, versus closing immediately when a client exits cleanly (at least, I'd assume so).

I'm not 100% sure; this is just "informed rambling." :)


That's fine- that is useful in giving me some leads to explore in understanding what I need to do to keep it tidy and clean. It's also possible that it may be complicated due to the fact that I'm basically using a library to emulate a UNIX tool on a Windows environment.

I did wonder about making the library keeping a static list of objects created so it can be accessed and told to zap them all if I needed to...

[ShadowOfGed]

You're opening an SSH tunnel? Bingo, you're probably launching an ssh process which will stay open once the parent process dies. Look for it in the task manager.

You probably have a couple creative options in this case. One, the hideous solution, is to write yet another application to monitor state, so it doesn't get taken down when your main app crashes. This is fugly.

Another option that might work is this:

When SSH forwards TCP connections, it behaves interestingly. If you open a shell like this: "ssh -L port:remotehost:remoteport user@host", you'll gain both shell access to the remote machine as well as having the desired TCP port forwarded. Now, if you open a TCP connection that goes through the tunnel, and exit the remote shell, SSH will not exit. SSH remains active until (a) the launched process is done running on the remote side and (b) any active tunneled TCP connections close.

You could abuse this fact through the following sequence of events:

launch SSH to open the tunnel
connect to remote side through tunneled TCP port
tell SSH to exit

Thus, SSH *should* remain running until your application closes its end of the tunnel. Thus, if your application exits or crashes, the only TCP connection through SSH will terminate, and SSH will be free to exit on its own, thus cleaning up after itself. However, this will not work if your application repeatedly opens/closes connections to the tunneled connection, as SSH will exit prematurely.

Just a thought.

This is what we call an elegant hack. Sometimes I feel brilliant. :p ;) :D

[Banana]

Geez, and all I wanted was some keywords so I can research the issue and do the work myself. ;) :p

But, seriously, thanks, man. This helps me a bit in understanding about how processes are handled by the kernel.

That said, I had a perfect chance as when I got to my Windoze box, it already had rebooted after update. With a clean slate, I took a careful look at the process list as I went through the motions and three-finger-saluting the application- Apparently, I may have been confused last time because 1) I didn't see any separate process being created when connection was created, 2) it cleaned up after itself even with the three-finger salute. It's quite possible that the problems I had actually is somewhere else and not with the library.

Let's check if my conceptions are correct:

1) When I instantiate a object from a library, that object is still within that application's process, and does not necessarily spawn a process of its own, unless the object was a kind of monitoring utility or something similar to that?

2) Libraries generally doesn't spawn processes on their own.

Correct?

[ShadowOfGed]

Let's check if my conceptions are correct:

1) When I instantiate a object from a library, that object is still within that application's process, and does not necessarily spawn a process of its own, unless the object was a kind of monitoring utility or something similar to that?

2) Libraries generally doesn't spawn processes on their own.

Correct?


Yes. Instantiated objects are always within the memory of that process.
Mostly, unless that library explicitly creates its own process to handle something externally. This, on occasion, is necessary. There could be a variety of reasons; it just depends on the library. However, if a "library" does this, the creation and management of an external tool should be transparent to you, as a user of said library.

[Banana]

When SSH forwards TCP connections, it behaves interestingly. If you open a shell like this: "ssh -L port:remotehost:remoteport user@host", you'll gain both shell access to the remote machine as well as having the desired TCP port forwarded. Now, if you open a TCP connection that goes through the tunnel, and exit the remote shell, SSH will not exit. SSH remains active until (a) the launched process is done running on the remote side and (b) any active tunneled TCP connections close.

Now I'm glad you mentioned that earlier else I'd have gone insane. :D

Apparently, I need to figure out how to recover from a lost connection, and the current solution is just to quit the application, which is kind of fugly. I can reset the object, but this doesn't reset the forwarded ports- e.g.

Working connection:
$ netstat -p tcp

Me:XXXX localhost: 3000
Me:3000 localhost: XXXX
Me:YYYY remotecomputer:ssh

After closing the object from the library, but not quitting the application:
$ netstat - p tcp
Me: XXXX localhost: 3000
Me: 3000 localhost: XXXX

After quitting the application:
$ netstat - p tcp


The ssh connection is now gone, but the forwarding seems to be orphaned until the application has been quit.

Thus two questions:

Can I just tell the SSH to re-use that port?
Is there a special code to force closing the connection; the library has functions to call delPortForwardingL, but as far as I can ascribe, those doesn't work, whether because of the application still using it, or because the developer was a moron, I do not yet know. :)