Matrix Speed questions

Greetings, I am still working on trying to port my critical office management system to Pan X. The biggest hurdle currently involves matrices and the speed of refreshing the information in them.

The main screen is a calendar with a column for each day, each having 2 small matrices displaying an action flag on/off, and transaction status flags. The main matrix adjacent displays names for calendar times. The names are done with lookups matching an ID number in an array for that day to a patient info database. In Pan 6 jumping in and out of the calendar to other screens or moving fwd/back for different weeks, the entire calendar redraws so fast it can’t be timed (there are a bunch of other displays in the header and footer, as well). This is on a 7 year old 21" iMac. BTW, opening the 15 database set takes a couple of seconds. In Pan X it takes 20 to 30 seconds before the beach ball stops spinning and everything is open on a sample set of three main files.

When I set up a sample matrix for one day in a Pan X mockup, the refresh is painfully slow. One matrix takes about 6-8 seconds. When I had seven side by side, and tried to space them using the space function, it took so long I went to make popcorn after watching it move them one pixel at a time for 2-3 minutes.

This is unacceptable, of course. I am using the same data and lookup strategy in the matrix frame object that I used in Pan 6. I tried a different way to have a formula do all the calculations and make sets of data arrays to simply display straight down the matrices, but that didn’t speed things up at all.

Am I missing something? Is this how slow the matrices are compared to the old version? Would a new computer help dramatically? Does anyone have any hints on setting up the frame object formulas or something that would make them become lightning fast?

Sorry, I know there are several questions buried in there, but if I can’t get the speed up, I’ll have to stick with Parallels and keep my Pan 6 running until I retire.

Does your 7 year old 21" iMac have a spinning hard drive, or an SSD? I can’t speak to matrixes specifically, but in general modern versions of macOS and modern Objective-C/Swift software do not perform very well on systems with spinning hard drives.

Scott - You don’t say how many records are in your database, but I have found Matrixes and Text List Objects to be incredibly slower in Pan X. A database with 100k records can search as fast as I can type in Pan 6. In Pan X, I have to pause between letters when typing a name or the search is interrupted because the object can’t keep up. I’m hoping an upgrade will address this issue as, for now, it appears to be unacceptable with large files.

Panorama 6 Matrixes and Text Lists didn’t even include search capability. If you are using old Panorama 6 code to search a matrix or text list, I recommend that you switch over to the new native search capabilities of these objects.

That sure wasn’t my experience.

I can’t use the Panorama 6 code in Panorama X. I was using livequery with a text editor object to update a list in Panorama 6. I am using the built-in _liveSearch with a text editor object to update a list in Panorama X. Both use “update variables with every key”. I have tried many of the suggested ways on this list to speed up searching with a Text List in Panorama X. I have read the user docs on these objects many times. It is still very slow.

A Pan 6 database with over 18,000 records shared on Enterprise using livequery can update the text list as fast as I can type, The same database on single user Pan X, using _liveSearch, requires me to wait between letters to catch up.

If there is a way to speed it up, please tell me.

I have been constructing text lists to display arrays, which can be rebuilt quite fast in response to searches. I posted a sample database with US zip codes (42,522 records) in the database exchange showing this approach. (This one searches only when return is pressed.) The arrays can also be sorted very fast on any column with a custom statement (ARRAYSORTBYCOLUMN), which is also in database exchange. I have not generally updated every key, but updating most keys works with well with a language database I have with about 12,000 records (and a lot of fields).
I have not tried using matrices in a long time, but I recall finding that they were suitable only for relatively small amounts of data. Any the array method has in my experiene performed better than linking the text list to the data via Database Navigator for databases above a certain size (I can’t say where that crossover is, though.)
I don’t know if using arrays will solve your issues, but they might be worth considering.

Jeff - I’m not searching another DB for this task, although I use LiveClairvoyance to search for names in a different database with only a few thousand records. As you say, it loads the names as fast as I can type on Pan 6.
In this case, just testing phase one to see if I can make it work. The matrices are in the main window in the Calendar DB. The DB has only a few fields that hold 48 element arrays. The Name Matrix’s job is to find the date corresponding to its position (ie third day of the week being displayed) and take the array and display it in the matrix by doing a lookup into the Patient DB to fetch the name that corresponds to the Patient ID # stored in the array. I hope this makes sense. Other matrices simply display the array for that day - such as invoiced or not gets a checkmark or nothing. These are dead simple displays.
From the old days I learned to optimize my DB’s by sorting the newest info to the top and arranging the fields so that the most searched ones are right at the beginning. When looking at the calendar data, we are usually only grabbing data in the top few records corresponding to the most recent week or two. So that should be helping.

Jim, the 21" computer at work has an SSD and the 27" at home, where I do most of the development work, has a 3 TB Fusion Drive. I assume since I use Pan X frequently for some basic utilities that the app and files are stored in the SSD section of the drive.

But once it’s loaded into RAM, that shouldn’t make too much difference in performance, should it? Only in saving or initial loading the files. Even in a sample file where I use a procedure to get one array from one specified date and display it in one matrix, it takes that long to refresh.

Following CooperT’s suggestion, I see that Text Lists are dramatically faster. I’ll see if that would do the trick.

Unlike with the same database clairvoyance between different databases in not yet supported in Panorama X. In the online Help documenting, not yet ready for all, version 10.2 the penultimate unsupported Pan6 field property, duplicates allowed or not allowed, obtains support. Those interested should read its fine print once available. But still no other database clairvoyance. Jim has stated it likely will be supported some day. Implementing it robustly with adequate error handling must be non-trivial. It’s apparently been low priority due to low demand. So if it’s valuable to you speak up. I’ve found less elegant ways to accomplish what it had done for me in Pan6, so don’t miss it… much.

Just to be clear, LiveClairvoyance was not the same thing as Clairvoyance in Panorama 6. It was a custom statement built around the SuperArrayBuild statement. It was used in conjunction with text editor and list superobjects to do the sort of searches that you can do in the Help file in Panorama X. With each keystroke, the list is updated to show only the matches.

Panorama data is loaded into RAM. But the program code might not be. The way Objective-C and Swift work they can be swapping code and system resources in and out. Also, macOS uses VM (virtual memory), so RAM can be paging back and forth to disk depending on the stress on the system. There’s no guarantee that something is “in ram” the way it would have been on systems 20 or 30 years ago.

The functionality of this statement is now built in to Text Lists and Matrixes, so a separate statement is not needed. It was going to be a considerable amount of work to duplicate the liveclairvoyance statement in Panorama X, and it would have not worked as well as the functionality now built in to lists and matrixes, so it was removed.

They will be somewhat faster because of the simpler layout. FYI, however, under the hood, Text Lists and Matrixes are built on the same code. And in fact, at the next lower level, Text Lists, Matrixes, the data sheet and view-as-list forms are all built on an Apple technology called NSTableView.

I would recommend switching to “update variables most keys” (that was recommended in Panorama 6 also). Also, make sure that background search is enabled in the list or matrix object. This is the same technology used in Panorama X’s Find/Select dialog. The live preview in that dialog sheet is implemented with standard Panorama X form objects – there is nothing about that dialog sheet that you cannot reproduce yourself. I put a lot of work into optimizing that code to make it as fast as possible. Months of work.

Panorama 6 could never search large databases as fast as you could type. Instead, it would abort the search when a new key was typed, and start over, giving the appearance of responsiveness. If set up properly, Panorama X will do the same thing, in fact better, because it has true multi-threaded code, which Panorama 6 never had.

Thanks for all the tips and discussion. Rather than have extensive lookups and if/then choices in the matrix object as in Pan 6, I had a procedure build and modify the 7 arrays for the calendar and then simply display the arrays in text lists. A preliminary trial with matrices shows marked speed improvement close to what I am used to. Not sure if I made any other changes, but it’s working.

I’m wondering if the text lists could have a fixed number of rows or fixed height similar to the matrix options. I have a number of really tightly organized forms in Pan and Apple Pages for my business. Over the years I’ve had to rework them all when Apple makes slight changes to some of the fonts for a new OS upgrade - even standards like Helvetica. I would hate to have to have that happen to my text lists and suddenly all the time slots on the calendar don’t line up with all the other graphics on the form.