Panorama X memory usage

Panorama X
1

I’ve been running some performance tests on the lookupselected( function and, according to Activity Monitor, Panorama X uses an enormous amount of memory, even for what seem to me to be relatively modest tasks (but maybe they’re not). Here’s a screen shot of Activity Monitor towards the end of a test where 16,000 records were looking up 8,000. The two databases have identical content, so the number of comparisons will be the minimum possible. No other apps were open.

The tabulation shows Panorama X using 22 GB of memory while the display at the bottom shows that almost the whole of the 8 GB of RAM is being used. Several questions come to mind:

  1. Is the Activity Monitor reliable or meaningful?
  2. Does Panorama X swap in and out of RAM when it nears the limit?
  3. What is actually going into all that RAM - seriously 22 GB?
2

Not really.

Did the memory usage go down once the test was finished?

Panorama doesn’t do any explicit memory management, but the system does. We’ve come a long way from the bad old MacOS manual memory allocation days. Swapping is handled by the OS in close cooperation with special features in the Intel processor. It’s completely invisible to individual applications. In fact, applications don’t even know the real memory addresses of the memory they are using.

So the answer to your question is sort of yes – certainly memory could get swapped out. But this could happen with Panorama 6 on OS X also. Either way, the application itself isn’t even aware this is happening (if it is). That is all “behind the scenes”.

As Panorama performs a task, it is continuously allocating memory then releasing it. This is the nature of Objective-C, you can’t write an Objective-C program that doesn’t do that. But the memory isn’t released immediately (obviously, or why allocate it in the first place).

For contrast, right now I have quite a few web pages open in Safari. Activity monitor says that just ONE of those pages is using 1.46 Gb, and another is using 1 Gb. This is beyond Safari itself, which is using a half gig.

I notice that you aren’t displaying the Real Memory column. On my system, this value is often quite different than the Memory column, sometimes much smaller, sometimes much larger.

However, all this needs to be taken with a grain of salt. As I mentioned, Objective-C (and Swift) programs tend to perform thousands or even millions of memory allocations to perform a task. Panorama X is almost certainly doing millions of allocations to do your lookups. Most of these are probably tiny allocations, but they add up (and I think allocating 1 byte may count as a minimum of 1k or maybe even 16k). There are so many allocations, I’m pretty sure the system is not accurately tracking them for Activity Monitor because that would incur a huge performance penalty, especially if tracked on a second by second basis. So I’m pretty sure the numbers you see in Activity Monitor are approximations. As far as I know, there is no way for Panorama itself to determine it’s actual memory usage (I’ve looked, though I won’t 100% guarantee that I just didn’t find it).

It’s pretty hard to get OS X, or any modern operating system, to NOT use most of the physical RAM. It will try to keep the RAM as full as possible at all times. There’s no point in having empty RAM, so the OS X memory manager makes sure it is all in use if at all possible.

3

Yes.

Thanks for all the other info. Altho I’m sure I’ll lie awake tonight wondering what the OS X memory manager finds to fill up the RAM :slight_smile:

4

It’s possible that there is some opportunity to improve Panorama in regard to short term memory usage during intensive tasks. There are a couple such tasks that release memory as they go, but most wait to release until the task is completely finished. In some cases I may be able to release some memory earlier. I’ve made a note to investigate this at some point in the future.

I’m not sure if this will make much difference, but it might for large databases with very large tasks. The task you are doing, a formulafill with a lookup( is probably a worst case as far as using temporary memory goes.

5

I have two Pan X databases each with ~500,000 records, 39 fields, ~40MB each. I attempted to do a lookupselected - about 50% of the records selected in each, and got an “Application Memory full” message showing Panorama using 123 GB!
I never once had this occur in Pan 6 under virtually identical scenarios.
I see others have raised this issue before, back in Aug, 17. Any updates on releasing memory before the end of the tasks?

6

Back in September 17 I made the change to release memory after each record when doing a lookup. However, this change had to be removed, because it was causing crashes, which is of course worse problem. So unfortunately there is not a simple fix for this situation.

7

Thanks for the update Jim. As a work-around I exported both identically sorted databases as txt files, opened each in Excel, then did a simple cut and paste to move the field content from one to the other. I imported the amended txt file into my template and, voila, I had the data I had wanted to move using Lookup.
Now, if there were only a way to copy and paste populated fields between Panorama databases…

8

Not sure how this would work for you but maybe worth a try. If you have both databases sorted and selected the same way with the same field name used in both databases, this should gather all the values from the first database and transfer their values to the second database replacing anything currently existing in that field

let theCommonField="MyField"
let fieldValues=""
arrayselectedbuild fieldValues, cr(), "",chr(1)+
    datavalue(""+theCommonField)
setactivedatabase "Second Database"
field theCommonField
formulafill array(fieldValues,seq(),cr())[2,-1]
setactivedatabase ""

Note that I have added a chr(1) character to the values of each record in the field so that we also capture records that are empty in that field. The formulafill then eliminates that character before adding the value to the field. Replace “MyField” with the field name you need and “Second Database” with name of your second database. Oh, yes, try this out on backup copies before committing to it fully.

9

Robert,

A few years back I wrote an Accelerated Lookup procedure which theoretically performs a lookup between two databases of any size and essentially has no memory limitations. It’s available on the Panorama Database Exchange site (accessed via the Help menu).

Unfortunately it incorporates a silly error in the reading of the input parameters but this is easily fixed. Download the procedure and then replace lines 92 to 99, which read:

mgKD2 = parameter(5)
    if mgKD2 = ""
        returnError "You have not specified the fourth parameter, the target database field from which the information is to be retrieved."
    endif
mgKD1 = parameter(6)
    if mgKD1 = ""
        returnError "You have not specified the fifth parameter, the primary database field which must match the target database field specified in the sixth parameter."
    endif

and replace them with these:

mgKD1 = parameter(5)
    if mgKD1 = ""
        returnError "You have not specified the fifth parameter, the primary database field which must match the target database field specified in the sixth parameter."
    endif
mgKD2 = parameter(6)
    if mgKD2 = ""
        returnError "You have not specified the sixth parameter, the target database field which must match the primary database field specified in the fifth parameter."
    endif

and all should be well. The procedure is thoroughly self-documented and you should be able to work out what’s going on if you’re interested. For very large files it can be a lengthy process but it has a built-in monitor which gives you an indication of where it’s up to.

The limitation for you is that it performs just a straight lookup, not a lookupselected so you will have to isolate your selection manually.

I haven’t got around to modifying the file on the Database Exchange due to a combination of laziness and lack of time. Also, I store my file on Amazon S3 which is about as user-friendly as a great white shark.

10

Thanks Gary! I’ll try this out and let you know the result. (I’m constantly amazed by folks’ generosity in the Panorama community.)