Some progress and perhaps some information:
I couldn’t get the lookup( to work so I replaced it with a record-by-record process and I’ve progressed as far as line 888 by dint of the insertion of about 25 message statements, some of them for every iteration of a loop.
Here’s the response to an info("databasename") message - it explains to some extent why the open database isn’t open - it doesn’t have a name. It has on occasion been referred to in an error message as “Database Null”:
I’ve finally got the procedure to run to the end (still riddled with message statements), but I’m left with an open Tickers database which is frozen, forcing me to quit.
En route to where I am now, I had this interesting situation:
Logical I suppose if one of the two Tickers databases doesn’t have a name in Panorama X. But should macOS allow two files with the same name? Is that perhaps part of the problem?
Tickers has many fields and loads of data so I’m not prepared to rebuild it brick by brick but I did export it as a text file, dumped the original and imported the text file to a new database. Didn’t make a scrap of difference. Also quit every other application with equal success. Any suggestions? Please, pretty please?
55 crashes in January, all in the last 8 days.
Fifty of these crashes have no Panorama code in the stack trace, only Apple code. Only five have any Panorama code in the log. A stack trace with no Panorama code in it has zero diagnostic value. Usually only a small percentage of stack traces have no Panorama code. This usually means that some code went way off the reservation, but didn’t crash immediately. It’s kind of like the perfect crime, leaving no evidence behind. (Occasionally it means there is a bug in Apple code – it does happen.)
Update: Looks like there are another half dozen or so crashes since I last updated the crash report database earlier today. All of these are only Apple code.
The closedatabase statement has a completely different internal path than the closefile statement. If you’re worried about which database you’re about to close, you can always just use the window or setactivedatabase statement just before the closefile statement.
Actually, the closedatabase statement is a recent addition, so it’s not nearly as battle tested over years as closefile. And it was designed for closing other databases, not the current database, I’m not sure if I ever even tested it for closing the current database.
I have a radical suggestion for something relatively quick to try – remove ALL closedatabase and closefile statements from your code. You can just manually close the databases at the end. Perhaps that’s not an acceptable long term solution, but it will identify if closing databases is the cause of the problem. I assume the databases aren’t so large that they can’t all be open at the same time.
You never mentioned a problem with lookup before.
It really feels like you are attacking this problem piecemeal. You’re never going to fix a 900 line program that way.
First of all, do you really have a task that requires 900 lines of code to run as a continuous block? In my entire career I’m pretty sure I’ve never written a block of code anywhere near that large. Figure out how to break the task into separate parts. Get part 1 working in a way that you can then skip to part 2 and debug that without having to run part 1 every time, then continue for part 3, 4, etc. It’s a lot easier to get separate small components working than to get one huge component working. Panorama X contains almost 200,000 lines of Objective-C code, and 60,000 lines of Panorama code, but there are very few subroutines over 50 lines, and only a handful over 100 lines.
Secondly, you’ve never mentioned using the most powerful tool that Panorama X now has for debugging, the Debug Instrumentation system. Using this system is kind of like building a machine out of clear plastic so that you can actually see all the moving parts as they are moving. If you do a careful job of adding instrumentation to your code, the source of an inscrutable problem will often jump out at you. I would really recommend that you carefully watch the video session on this topic, then take the time to add good instrumentation to your code. To add good instrumentation you’ll need to carefully think about your code, and sometimes just that thought process can expose issues.
One last kind of wild thing you could try – downgrading back to Panorama X 10.2 b24. Though the changes between b24 and b25 appear minor, there was actually a huge change in the tooling used to build Panorama X. Up through b24, Panorama was built with Xcode 8, but b25 was built with Xcode 12, which is 4 years newer. This is a big change. I recently discovered at least one crashing bug which was caused by this change (and which I haven’t been able to track down yet). Your crashes do not fit the signature of this bug, but perhaps there is another new problem I’m unaware of so far. If you don’t still have a copy of b24, you can use the Check for Updates window to download it, there are instructions for downloading older versions here.
One last tip – be sure to take breaks!
Is it possible that the Apple crashes are related to the existence of two files with the same name, as mentioned above?
I’ll try that although many of the Panorama X freezes and error messages have occurred at window and setactivedatabase statements - almost all related to the Tickers database.
As noted above, I get freezes and error messages triggered by a range of statements. There just didn’t seem to be any point in reporting all of them given that they all appeared to have the same issue - the non-existence of the Tickers database.
This is an excellent suggestion which I’ve had in mind because it’s what solved a similar problem with a different procedure in the the same database. And this procedure is modular, with nine discrete components.
However, inserting stop statements to halt the procedure at selected points, I’ve tried running just the first module (no fails) then the first two in sequence, at which stage I get a fail (defined as a freeze or error message) in the first module. Then I can sometimes run them again with no problem. But maybe breaking the procedure into several independent procedures will help.
I’ve never used a debugger but I suspect it’s now time to do so.
An interesting suggestion but, if it worked, I’d be stuck at that level of development, never to experience the joys of higher versions. Actually, as I write, I see that Pierre is trying that.
Thanks Jim. I’ll keep trying. And I do take breaks.
Yes, I did try using zlog statements scattered throughout the offending proc, but I found that when PanX crashed, there was no info in the log - and yes it worked in the absence of crashes. I figured, maybe incorrectly, the the new debug system didn’t work when a crash occurred. Could the debug info have been buffered and never made it out the door?
I also downgraded PanX to b24 but that made no difference. I’m now back to b25.
If you are looking at the log file, then yes, that is definitely buffered and will not be complete if there is a crash.
In this situation, what you want to do is run Panorama X under Terminal.app. The log in Terminal.app is not buffered, and will be 100% complete right up to the point of a crash. The Debug Instrumentation page explains how to run Panorama X under Terminal.app. It’s quite easy, but does require a short setup process, and does require you to launch Panorama a special way each time.
Unless you’ve installed a case-sensitive file system on your Mac, you do not have two files with the same name. And you’d know if you had installed a case-sensitive file system.
Your screen shot shows that you have two windows named Tickers. That does not mean you have two files named Tickers. A form window can be renamed to anything you want – one of those could even be some other database.
FYI I’m not talking about a debugger. But you’ll see what it is when you watch the video.
I wasn’t suggesting that this would be permanent, just that it might indicate whether this was something that had changed in the b25 version. Apparently it isn’t, so this is a moot point.
Ok, now I’m guessing that this is originally Pierre’s database, and Pierre’s 900 line program, and you are helping him with it? That’s fine, of course, I’m just trying to understand the players. It’s been a bit confusing with both of you chiming in, but if this is actually Pierre’s database that makes a bit more sense.
I don’t understand your reference to a form window. None of the databases in question has any forms. And when I hit the F3 key, I definitely have two Panorama X databases with the Tickers name at the top. My thought was this would worry macOS.
Well, not exactly, it’s his program but I wrote it - he’s still on his trainer wheels. It’s part of a large project.
I shouldn’t have restricted it to form windows. Any window can be renamed to any name with the windowname statement. For example, here is a data sheet.
By running the code:
windowname "Fake Window Name"
I now have this window:
If I already had a database open with the name Fake Window Name, it would now appear that I have two data sheets open with that name.
I’m not saying that you necessarily are doing this, just pointing out that two windows with the same name doesn’t mean that two files have the same name.
F3 is Expose, correct? I don’t usually use the Fn keys, but it appears to be expose on my system.
So when you way you have two “databases”, I think you are meaning to say that you have two data sheets with the Tickers name. Assuming you didn’t use the windowname statement, this would probably mean that at some point Panorama got confused and allowed the same database to be opened twice in separate internal (in-memory) data structures. This is not supposed to ever happen – the first thing the open database code does is check to see if the database is already open. But for some reason this check isn’t finding a match.
I tried experimenting with the closedatabase and closefile statements to see if I could duplicate this problem, but I couldn’t get either of these to fail in this way (or fail at all). I did verify that the closedatabase statement works fine when used with the active database.
So I think something in your 900 lines of code is messing up the internal structure that Panorama uses to manage open databases.
You should be able to check the number of Tickers data sheet windows with this formula.
linecount(arraystrip(arrayfilter(info("windows"),cr(),{?(import() match "Tickers",import(),"")}),cr()))
You could create a mini-procedure at the end of your big procedure like this:
checkTickers:
let tickerWindows = linecount(arraystrip(arrayfilter(info("windows"),cr(),{?(import() match "Tickers",import(),"")}),cr()))
if tickerWindows > 1
message "Extra ticker at "+parameter(1)
stop
endif
return
Then, at various spots you could call the subtoutine this like this
shortcall checkTickers,"spot ABC"
...
shortcall checkTickers,"spot DEF"
...
shortcall checkTickers,"spot XYZ"
As soon as more than one Tickers window is detected, the procedure will stop and display a message indicating where the problem is. Whatever the problem is, it must be in the code above where the shortcall statement was. Since apparently Panorama’s internal memory structure is compromised, I would suggest quitting and relaunching Panorama after each time you run this test.
You can start with just a few shortcalls to this mini-procedure, then keep adding more as you learn what the target area is.
It’s possible (even probable) that the actual problem is caused before the duplicate window symptom appears. So this probably won’t find the exact line of the problem. But the problem is definitely not caused after the duplicate window shows up, so this should definitely narrow down the scope of the search for this problem.
My heartfelt thanks for the time you’re putting in on this. Of course, if the end result is the discovery of a big bad bug, it will all have been worth it.
On your final point,
I think you’re spot on. I’m now running a procedure which calls (as separate mini-procedures) just the first two modules of the big procedure. The second mini-procedure (.UpdateTickers3) runs successfully on its own but, when both are called in sequence, the second keeps failing at a lookup("Tickers" ... statement despite numerous amendments to the first, finally producing this gem:
I’m taking a break from it today.
It seems like perhaps somehow you are winding up with a window that doesn’t actually have a Panorama database backing it. I have no idea how that could happen, but it seems like that could match the symptoms. Maybe somehow at some point the database is getting closed in memory but the window is not actually closed. Then if you open that file again you wind up with two windows.
You’re probably all fed up with this saga by now but this is really for Jim’s benefit.
I have now subdivided the large distribution procedure into 11 separate procedures. If I run the 11 procedures independently by choosing them from the Action menu one after the other they run perfectly. If I run a procedure which calls them in turn I get several intermittent and not always repeatable fails, almost always with a frozen database which has to be quit and then triggers a crash
All of my recent crashes (maybe 30 or more?) have been Apple-inspired after Panorama X freezes, mostly due to Panorama X’s inability to recognise an open database, and has to be manually quit.
The superalert statement with a timeout of one second can sit there forever unless I put a delay statement between it and a preceding closefile and sometimes it does it anyway.
The following is a common chain of events if all modules are run in sequence:
superalert statements close - have to be closed manually. Quit and reopen manually.
Replacing superalert “End of Module 8”, {timeout=1} with message “End of Module 8” solves this problem (every module ends with a superalert so I know where it’s failed). There certainly appears to be a problem with the superalert statement.
The sequential call procedure is now running successfully three times out of four after inserting delay statements all over the place. That’s great progress compared with where I’ve been.
Oh, and a touch of deja vu - the old issue of save followed by closefile has re-emerged. I have to separate them with an nop.
The most helpful idea I have is the the dog is inserting some malicious code so you will come spend more time with him. But I can say, for the little that it may be worth, that I am very suspicious in these situations, and I have been in them, using alerts. My choice would be to us the logging system with zlog statements to monitor the progress of the procedures and possibly report errors.
I really wish you would have tried this suggestion I made a few days ago. Based on some tests I just did, I think it would eliminate all of the problems in your code. Basically, if you close a database and then immediately open it again, the open doesn’t work.
This may be difficult or impossible to fix. The closefile and closedatabase statements work by telling Apple’s code to close the document. I believe the Apple code does this asynchronously, in other words, the Apple code returns control to Panorama immediately but the “close” operation continues in the background. For other asynchronous operations like saving and printing, Apple provides a mechanism to notify the original program when the task is finished. When running Panorama code, this notification mechanism allows Panorama to stop and wait for the task to finish before proceeding to the next statement. But there is no such notification mechanism for close. Apple seems to assume that they can take whatever amount of time they want in the background, and normally they would be correct.
For the tiny database I was using, I found that the problem went away if I used a wait 1 after the close operation. A delay of 0 wouldn’t work. But I think there is absolutely no assurance that 1 is enough for any database, it probably isn’t. I don’t think there is any way to know what a sufficient time is. I do not recommend putting wait statements after your close statements.
What does the doctor say when you tell him “it hurts when I do that”? He says don’t do that! You haven’t explicitly told us that you are closing a database and then turning around and opening it again, but you are doing that, right? I think the best solution is to not do that. At the start of your task, open the databases you need. Leave them all open until the task is complete. If for some reason you don’t want the windows visible, use makesecret. But don’t close them until the end.
I believe that the only time I was doing that was when Module n would close all open databases and Module n+1 would open one or more of them again, with each module procedure called in turn.
The locations of the fails I experienced were highly varied and inconsistent with no guarantee of repeatability. With the procedure left open, the problematic code line was sometimes highlighted but most times the procedure just displayed its first page with no line highlighted, even though there was an error message in the dialog line at the bottom.
I have taken your doctor’s advice - I’m rebuilding the procedure (much reduced in size) module by module and thoroughly testing each one as I go. Like Frank Sinatra, I have high hopes.
And I’ll do that.
That would do it. The fact that the code is in different “modules” doesn’t mean anything to Panorama’s runtime engine, which simply runs each statement as fast as it can. So the first statement of module 2 will run just a few milliseconds after the last statement of module 1. The effect is pretty much the same as if the two statements were right next to each other in a single procedure.
I didn’t make that clear - Module n and Module n+1 are separate procedures. A master procedure calls each of them in turn so Module n has disappeared (or has it?) before Module n+1 is executed.
I guess I didn’t make it clear either. It makes no difference that they are separate procedures. Panorama just executes statements one after another. There is no significant delay in calling a separate procedure.
Module n has disappeared (or has it?)
No, it hasn’t disappeared. It’s still in memory. When you open a database, all of the procedures (and forms) are all brought into memory immediately, and they stay there until the database is closed.
Actually, when you call a procedure, Panorama makes a copy of that procedure in memory, so that if the database is closed while the procedure is running, the procedure can continue. It’s actually the copy that runs, not the original procedure. But making that copy only takes a few tens of microseconds.
I reckon I’ve learned more useful Panorama X peculiarities and tweaks in this topic than in the last couple of years.
I reassembled the procedure and, taking Jim’s advice, delayed all file closures until the end and it ran without a hitch.