[WriteLog] Software Memory Leak (comments)

[Dick Green]

Nice explanation, Rick. I would add these comments:

1. Not all memory leaks are the fault of the application -- the operating
system can forget to release allocated memory, too.  Having witnessed
countless hangs and crashes of Windows 95 and 98, as well as inexplicable
episodes of very slow operation that can only be fixed by rebooting, my
conclusion is that the memory manager in those versions of Windows has bugs
and/or design flaws. The memory usage utility often indicates that the
percentage of allocated memory is near maximum, regardless of what's
running. My impression is that these versions of Windows don't do a very
good job of releasing large amounts of unused memory (if they do it at all.)
With only small chunks of RAM available, the system tends to thrash. So far,
I've not seen anything of the sort under Windows 2000 -- memory utilization
seems quite reasonable.

2. The operating system's scheduling and paging algorithms have enormous
impact on performance. The scheduling algorithm determines when each process
or program gets CPU, memory and I/O resources, while the paging algorithm
determines when and which chunks of memory get paged to disk. As you might
expect, these algorithms interact and things can get mighty complicated --
for example, when it's a particular program's turn to run it would be nice
if most or all of the pages it needs are in RAM. This can only be achieved
through some pretty hairy optimization. My impression is that these
algorithms are relatively primitive in all versions of Windows, with the
possible exception of Windows NT/2000. We had much better performance with a
heck of a lot less memory back in the early days of time-sharing (early
1970s.) For example, the DTSS operating system (Dartmouth Time-Sharing
System) comfortably supported 250 simultaneous mainframe users with only
about a megabyte of memory! You can bet that the scheduling and swapping
algorithms were extremely well designed and implemented in that system. The
Honeywell (ex-GE) mainframe we ran on had a much slower CPU than today's
PCs, but it did have better memory and I/O design -- separate memory and I/O
processors offloaded the CPU. I think the I/O bus was faster, too. It's true
that today's programs require *much* more memory than the simpler
applications of the past (a little of that is due to sloppier coding), but
there's no question in my mind that Microsoft doesn't have the best OS
designers in the world.

I'll comment on the problem that raised this issue in another email.

73, Dick WC1M

> Reply-To: <Rick.Commo@verizon.net>
> From: "Rick Commo" <Rick.Commo@verizon.net>
> To: "Writelog" <writelog@contesting.com>
> Subject: RE: [WriteLog] Software Memory Leak
> Date: Wed, 2 Jan 2002 23:12:41 -0800
>
> This is a multi-part message in MIME format.
>
> ------=_NextPart_000_0029_01C193E2.FA2623A0
> Content-Type: text/plain;
> 	charset="US-ASCII"
> Content-Transfer-Encoding: 7bit
>
> What Len is saying is quite true.  And you have to consider that
> Win2K with
> Win9x/ME are all "virtual memory" operating systems.
>
> Time for a 20,000 foot view of virtual memory..
>
> What does "virtual" mean?  Nothing more than the fact that when
> RAM is about
> to be used up the OS will take some that hasn't been used in a while and
> write it out to the "swap file".  This frees up that chuck of RAM
> to be used
> by an active program (process actually). The part of the OS called the
> "memory manager", along with some help from the hardware can make any
> section of RAM appear at any address within the 4GB range represented by a
> 32 bit address.  So if your hard disk is big enough to afford a 4GB swap
> file then you can have a full 32 bits of "virtual" memory in use at once.
> Even if your machine only has, say 128MB.  Having said that, let's make it
> perfectly clear that this would be a *very* ridiculous set-up.  But in
> theory it's possible.
>
> Let's look at the problem on a more practical note.   Let's say that you
> have 128MB along with a 256MB page file.  As the 128MB or real, physical
> memory is used up sections that haven't been used in a while will get
> "swapped out" (written) to the page file and the newly freed up RAM is put
> back into the service of a different process or program.  As a rule of
> thumb, try to have as much RAM as you can afford and make the
> swap file the
> same size or twice as large.
>
> If your disk often sounds like a popcorn machine then your computer is
> really hitting the swap file hard and you need more RAM.  Incidentally,
> there is a range of opinion as to what the ratio of swap file size to RAM
> size should be.  I use 2:1, mostly because that was a recommended practice
> that I read about somewhere.  Others can make good arguments to move the
> ratio up or down.  If you are using up what virtual memory that
> you have (in
> our example, 384MB) then you might be able to delay the agony by
> increasing
> the size of the swap file - but it will cost you in performance.  RAM is
> orders of magnitude faster than disk!
>
> With 128MB on a machine running Win2K Server, I often heard the disk
> swapping.  I might just do something as simple as activate a window in the
> background that I haven't used in a while.  I would often be subjected to
> 2/5/10 seconds of "popcorn popping" later before the window would finally
> pop up.  I increased the RAM to 512MB when prices were cheap.
> Now I rarely
> hear the disk swapping.  And things seem a lot faster.
>
> Now back to WriteLog.  WriteLog would probably have to be working one one
> *huge* log to push the limits of a reasonably set up machine.  What *is*
> more poisonous is what Len talked about.
>
> Imagine a program that asks for a chunk of memory as part of an operation;
> and that this operation is done over and over again.  The chunk of memory
> was only meant to be used temporarily, but the programmer forgot
> to release
> it in some other part of the program.  Bingo, you've got a memory leak.
> Over time this *can* eat up your RAM and swap file.  This is
> where Perfmon,
> the NT/WIn2K/WinXP performance monitor, can help you track memory usage.
>
> Hope this hasn't been too boring for folks.
>
> Cheers,
> -ricK7log