B26 crashed 5 times tonight

Sorry to report that I just had 5 crashes in a row with b26 this evening. I’ve just had one crash with b25 since February; it was really solid. I’m reverting to b25 for now. I do have some of the crash reports if that would be helpful for anyone.

Panorama automatically sends crash reports to our server, so you don’t need to send them. The automatically sent report contains additional symbol information that helps decode the crash report, they are completely useless without this. Crash reports can sometimes be very useful but often they aren’t. Unfortunately that’s the case here. Our records show you had 4 crashes tonight (the report is only sent in when you relaunch Panorama, perhaps you didn’t relaunch after the last one yet). All of your crashes tonight had the exact same crash pattern, and this pattern has occurred before - 19 times since last summer. It has happened with b24 (this same exact crash occurred on your computer on November 12), b25 and b26, but has only occurred on three users computers. Also, it appears that all 19 of these crashes have occurred on macOS 10.13.6, so this particular problem may be related to that older version of macOS. I guess the bad news is I don’t think reverting to b25 is going to protect you against this particular crash.

Fingers crossed, so far the number of crash reports for b26-4044 is very low, even taking into consideration the low number of people that have it installed. In fact the only crashes so far have been yours today, and one on Tuesday. There also was a b26-4029 release about two weeks ago that had a very limited release of 8 users, only one had a crash. So far this is by far the lowest rate of crashes for any release since crash reporting was incorporated into Panorama X.

Perhaps my crash reports then are not being received?

I also cashed yesterday while viewing a procedure. 6 databases just disappeared instantly.

The most recent crash report received from you was May 24 @ 2:35 AM.

The most recent crash report received from you was May 16 @ 3:19 PM.

So maybe there’s a problem. I crashed once yesterday and once today while looking at a procedure (b26).

It appears that you are not getting the memo. There is a lot of good stuff you should know about. I’d been keeping some crash notes on the server and I will now start documenting the crashes on my client for better understanding of what I am seeing.

Do you mean “now start documenting?”

Thanks Dave for the proper attention to what I write. :slight_smile:

Documentation of steps taken that can reproduce a crash is great, I love that. However, please do not send crash reports. Unsymbolicated crash reports are completely useless. Theoretically it would be possible for me to manually symbolicate a crash report that was submitted “ad-hoc”, but it’s a process that would take 30-60 minutes per report, and still be unlikely to be helpful. Panorama is designed to automatically submit crash reports in a specific format that allows symbolication in a few seconds, it all happens automatically. Even with symbolication the results are not always super helpful (though sometimes they are very helpful), but without symbolication it’s pretty much just an incomprehensible jumble.

I think it’s possible to configure a computer so that crash reports are intercepted before Panorama gets a chance to send them, and instead appear in a dialog on your screen. I think it’s some sort of system preference. However, I have never been able to figure out what that preference might be. It appears that very few users have their computer set up to intercept crash reports because we receive plenty of them. I’m not super concerned if a small percentage of computers aren’t submitting crash reports because almost certainly the same crashes are happening on other computers.

On thing that has been helpful more than once is a report that "I did this and that and there was a crash, this was on March 12 at about 2pm (or whatever date and time it was). Sometimes I’ve been able to find the automatically submitted crash report at that time and match it up with the description, and the combination has led to the solution.

I don’t know if you change anything on your system, but the provue server did receive a crash report from you today at 1:53PM. It looks like you were perhaps opening a database? Unfortunately this is one of the crash reports that doesn’t make a lot of sense all on it’s own.

Both crashes happened while adding records and typing into the Datasheet. 2 pretty close together. After the 2 repeating crashes, I then decided to do something different instead of it happening repeatedly. I entered some data, Saved, entered more data, Saved. It did not happen again if I saved often.

This was on the client on a non shared db. The server did also crash today but I do not know any corresponding data as to why or when. I re-started the server at 3:07 and it gave me the POSIX error. I restarted Panorama, then the server came up ok.

I am thinking that the POSIX error that happens when my server Unexpectedly Quits to be associated with the b26 release.

Further, to contradict my previous assertions, it appears that my server has not just ‘stopped’ while also remaining running. Paying more attention tells me that the ‘stopped’ assertions that I have previously made were all in fact Unexpected Quits. (I call any silent disappearance of an application an Unexpected Quit as that was the previous wording when Apple provided such.)

Rather than assume you noticed, the error refers to a TimeMachine backup as the location being accessed. I wouldn’t expect that to be deliberate. Nor would I expect that to be a desirable location for successfully running shared files.

While I did see TimeMachine noted in the POSIX error, do rest assured that I am not manually moving any shared databases to any locations.

The Shared DBs are no longer sitting in a Shared Database folder in the Applications/Panorama/Extensions/Enterprise/Public Folder/ etc and I’ve had no anxious extra moments to search their new location out. I’ve only used the tools as they were offered, to share a file. Yes, that drive is backing up via TimeMachine to a NAS named FasTrak but that should be an acceptable method to backing up a drive without negatively affecting Panorama.

Nah, I never suspected you’re deliberately doing any of that, but observing that it would surely be a bad location for a file being accessed.

My guess is that something in the OS handling of doing the backups has set this up. I’ve had instances in which wrong copies of files (not just Panorama) are in use. I’ve suspected aliases of being at least one source of these scenarios. It’s as if the OS maintains the alias’ connection to the wrong copy.

After quitting, I’ll discover that my changes don’t seem to have been retained. After some drive searches I often turn up a newer copy with my changes. :face_with_raised_eyebrow:

The error message you included is happening when Panorama X Server starts. It’s not really directly related to Panorama X Server though. Let me explain.

A few weeks ago Jim Cook reported to me that he had difficulty getting Panorama X Server to work on a particular computer. Before I had a chance to get back to him, he figured out the problem, which is a good thing because I would never have figured it out remotely. The problem was that he had installed another program that used HTTP port 8080 (in particular, CrushFTP, though any program would cause this problem). Since Paanorama X Server couldn’t grab control of the port, it obviously didn’t work. However, it didn’t really fail with a helpful error message.

So, I decided to add a check before launching the server to make sure that the requested port was not already in use (whether it is 8080 or whatever port you have described). With some research I discovered that the lsof shell command could be used to check port usage. So starting with b26, when you start the server, Panorama examines the output of the lsof shell command to check whether the port is in use. If it is already in use, it displays a helpful indication of what the problem is, then refuses to start the server. You can then either quit the other program that is using the port, or change the port that Panorama X Server is configured to use.

What I didn’t anticipate is that the lsof shell command might not work at all on some computers. Panorama X is only asking for information about a specific port. Here’s what I get on my system right now if I run this command. I’m currently running Panorama X Server on this computer so it shows that this is port is in use twice, for IPv4 and IPv6.

Last login: Sun Jun 12 14:59:14 on ttys003
jimrea@Jims-13-M1-MBP ~ % lsof -i:8080
PanoramaX 43108 jimrea   12u  IPv4 0x971cfe7b8bdb40cf      0t0  TCP *:http-alt (LISTEN)
PanoramaX 43108 jimrea   13u  IPv6 0x971cfe7b965df967      0t0  TCP *:http-alt (LISTEN)

Apparently on Robert’s system this shell command produces an error. I have no idea why, especially since the error message seems to have nothing to do with port 8080.

Robert submitted this error message privately to me some time ago, so I’ve made a change for the upcoming version of Panorama. Panorama still runs lsof when starting the server, but now it checks for an error. If an error occurs, Panorama just ignores it and proceeds to launch the server without verifying that the port is available (like it did for b25 and earlier). I don’t know of any other way to check for a port being in use, so this is the best I can do.

So, you can see why I say this message isn’t really related to Panorama X Server. At the time this error message appears, Panorama X Server hasn’t even been launched yet.

Robert, until the next version comes out, I think you’ll have to go back to b25. Or, if you can figure out why lsof fails on your system and correct, it, you’ll be able to continue to use this version. It might be illuminating to find out what happens if you run lsof in Terminal.app.

Automation-21:~ rameeti$ lsof -i:8080
lsof: WARNING: can’t stat() smbfs file system /Volumes/.timemachine/FasTrak._smb._tcp.local./31D1638C-8735-40EA-9413-84A8C88D6CB8/Time Machine BU
Output information may be incomplete.
assuming “dev=36001ec6” from mount table
PanoramaX 72460 rameeti 6u IPv4 0xa8708ea5f97fcd47 0t0 TCP *:http-alt (LISTEN)
PanoramaX 72460 rameeti 7u IPv6 0xa8708e9c5ff3a48f 0t0 TCP *:http-alt (LISTEN)
PanoramaX 72460 rameeti 22u IPv4 0xa8708ea5f97f02a7 0t0 TCP> (ESTABLISHED)
Automation-21:~ rameeti$

Interesting as my TimeMachine backups to my FasTrak NAS are working fine on this computer (though yes, I have had errors message about my TimeMachine backups on another device.)

Part of the answer/problem is probably that this server is running on 8080. My Enterprise server is running on 80.

It may be of interest to know that when I am attempting to see if a port is open AND if a service is advertising it availability on that port, I use https://canyouseeme.org/ Only if a service is running AND the port is open will I get the green light.

That’s only going to work for services that are accessible on the public internet. I need to check on the local machine.

Panorama X Server cannot run on any port below 1000. In fact, macOS does not allow any service to use a port below 1000 unless it is a daemon installed at boot time. So if you want to run Panorama on 80, you have to use Apache (or some other external server you have configured on your system) and use the CGI to forward to 8080 or some other local port about 1000.

It’s very interesting that this is issuing a warning but then is also including the information we want. If I could impose a bit further I am curious what the results would be if this is run on your system through the posixscript( function, like this:

When I said that my Enterprise server is running on Port 80, I am presuming such as that server is serving web pages successfully. When I added the Panorama X server to the network, I did have to change the port to 8080 for the Panorama X server. But perhaps my brain is foggy on a bit of this as something doesn’t jibe. I’m just remembering that I had to adjust the Pano X server to share files remotely on the same subnet as the Enterprise server.