Synchronization Problems with Remote Clients

We apologize in advance for the length of this post. The last couple of days have been challenging as we have incurred some data loss with our shared databases. Most likely we are doing something wrong here. Some simple data changes (e.g. updating a date field in a record) on Client A were not getting to Client B, after Client B synchronized the file. The data still was not received on Client B even after Client B quit PanX and reopened the file that should have received the data. Yes, Client B and Client A were connected to the server as indicated by the wifi icon on each computer. Periodic pinging of the server has also been included in our troubleshooting.

Some non-critical, new generation changes (a few lines of benign code) were not getting from Client A to Client B via the server through both a non-critical and subsequent critical new generation. To force the changes to be transmitted from Client A to Client B, we decided to take the file off-line and do a full strength Field Arrangement and Data Types new generation on Client A. When Client B logged back on, the changes (both updated records and additional lines of code added to a procedure) were indeed received on Client B. However, some data records added on Client B were erased sometime along the way when Client B shut down allowing a full strength New Generation to occur on Client A.

We will try to activate the server log as was previously suggested. We have not done so due to lack of time to view the video instructions from last spring, but we now see that we must do something different in order to help solve these problems.

One strange thing that we notice almost always: There is no wifi icon displayed on a form indicating that the file is not connected to the server. However, there is a wifi icon on the Datasheet related to the form. Because the File Menu shows the option of Disconnecting from the Server, and because we have been able to ping the server, we have believed that the clients have been truly connected to the server.

It is probably helpful to know, our clients are NOT connected via a local network, but all of our clients (4 in total number at this point) are connecting remotely. That probably adds another degree of uncertainty to the equation. I am wondering if it is possible that a bad wifi connection at a remote location would allow the lack of synchronization that we have been experiencing?

As always, we appreciate any suggestions or insights.

That should cause an error, not a silent failure.

Did you try Downloading Data on Client B in addition to synchronizing? That shouldn’t be necessary, but if you do that it does download everything from the server. Slower, but should definitely get all the data.

I’m living your life. Do not pay any attention to the WiFi icon as it will only give you comfort that will lead you astray. It only means that the file did connect to the server when it opened. If it loses the connection 5 seconds later, you still have the WiFi icon.

Do not pay any attention to the ability to ping the server. The server computer can be up and running but Panorama Server may have Stopped. Panorama is still up and operating but the server has Stopped.

Yes, when Client B loses connection, you can sometimes find yourself entering data and it will only be sitting there in B and not getting to the server. And yes, the data is easily lost if you are not adept at acquiring the data through a text file write to a local file, then an import later. It just is not pretty to recover that data.

When you see that you do not have the WiFi icon, then you are guaranteed to know that there is a problem. All of what I am experiencing is local so the remote is not adding to the issues.

As a single user using the Server shared file situation for testing and wanting to be aware of what the server situation is, I am very interested in figuring out why the server is stopping. I am very confident that some code stops the server. I have written code that I was able to reliably stop the server due to some part of the code that it did not like.

There are 2 ways to ascertain that you have a problem. One is to use a screen sharing progam and visually check to see if the server is Stopped. If so, Starting the server via the preference is easy enough. Then you can either restart the clients, or use the Disconnect from Server and then the Connect to Server. That gets you back where you need to be.

I have written a very short procedure that I was using for a while and I believe this is an acceptable way to ascertain if you db currently has an open session.

Local LStatus
CheckServerConnection "Automation 21",LStatus
Message ?(LStatus=-1,"Automation 21 is 
Connected","Automation 21 is not Connected")

I am remotely connected to three different servers, two of which are installed on Macs that are owned by my clients. I have found the server to be very stable. I do not experience the server stopping even with multiple users accessing it. The ability to tell if any individual client is connected to the server is another story.

I use code similar to Robert’s in procedures to check for connection before proceeding but the best visual read I have is to look at the preferences (PanoramaX menu-Preferences-Client tab) and see if the databases are actually connected. This was called Sharing Info in Enterprise.

FYI, the “source of truth” for this panel is the same as for the “WiFi” icon.

I’m suspecting that there is some bug that under some (as yet unknown) circumstances the WiFi icon fails to display at all. But it should always match the Preferences/Client panel. I will second Jeff’s suggestion that this panel is a good way to double check this. If this panel shows that the database is connected and there is no icon, that is a bug in the display of the icon.

The code that I posted is not good code. If I purposefully Stop the server, and then run the code, the 2nd message does not display informing me that the server is down.

More importantly, if I manually stop the server, then immediately start the server, the db will not have an open session (by design.) But if I then run the code above, it will respond that it has a connection (which it does, but not an open session.) If I attempt to add a record to that shared db, nothing happens. But if I then Disconnect from Server, then Connect to Server, all is back to normal. So while I was contemplating running the code above on a timer, what we all really want to know is if our shared db has an open session, not just if it has a connection to the server. Big difference.

This is where I am puzzled with your results. This should produce an immediate error on the local computer, and in my testing, it does.

In the Client preferences panel, you can specify whether you want an alert or notification when an error occurs. Which do you have it set to? If notification, are Panorama notifications set up properly in System Preferences?

Whereas I typically get a notification, in my most recent test, nothing happened. (My preference is set to Notifications.) When I pressed Return to add a new record, there was no response at all. As if I had not pressed the Return key.

Today and yesterday has been odd in that I have repeatedly been getting db not open but while I typically then go and Start the server (due to it being in a Stopped condition), it was already running. For some new reason, that was not the issue. Now I was just having to Disconnect from Server, then Reconnect to Server to get going agin.

Does this database have a .NewRecord procedure that is trapping the Return key?

What if you use the Add Record tool or menu item to add the new record instead of the Return key?

We continue to struggle with synchronization. We have moved two computers to a local network with the server to eliminate any potential wifi problems. We have some files that synchronize beautifully. We have some files that refuse to synchronize. We can manually download the data from the server related to the files that do not synch so we know that new records entered are being uploaded and stored on the server.

Regarding server connection indication, we have had cases where the server has stopped, but the Client/Preferences have indicated that the database is connected. In the same instance, the wifi icon on a form was absent while the wifi on the attached Datasheet was present.

This was discussed in detail on another thread here on the forum. Panorama does not have a continuously updated indicator of whether the server is running and active right this second. Both the wifi icon and the client/preferences panel indicate whether Panorama was able to connect when the database was first opened.

Yes, you already reported that. Please let me know if you discover any pattern to when this occurs. As I have already stated, they should always match.

We are making progress in isolating the Synchronization problems. We apologize for the delay due to the limited time we have to test. We can replicate the problem. It has something to do with the way we initialize our PanX enterprise software on launch. We have approximately 12 shared files. Several of these files have database relationships with other shared files. When we first launch our PanX software, we execute a series of statements for each of the 12 files. The following is an example:

OpenSecret “UnitMix”
SetActiveDatabase “UnitMix”
Call .Initialize

We use this method to initialize several file global variables needed in each file. We like this startup method, because it is effective in opening files in memory, initializing variables, and preventing screen flashing. However, we now believe that this method is also part of the problem.

Here is how we can replicate the problem. Client A is the primary computer, and Client B is the test computer. Client A makes a new generation on a file F. Client B launches our enterprise software and opens file F. File F is not connected to the server. However, other files on Client B that were not part of the new generation are connected to the server. If Client B quits and restarts our enterprise software for a second time after a new generation, file F will be connected to the server.

This unexpected loss of connection after a new generation is the cause for files getting out of sync. This is why we were noticing that changes in a new generation were not getting to Client B. This is also why adding records on Client B were sometimes not hitting the server.

Part of the problem has been the absence of the wifi icon on forms when we start our PanX enterprise software using the method described above. Almost always the wifi icon is missing on a form, but if the connection to the server exists (when there has not been a new generation), there is a wifi icon on the attached DataSheet which coincides with a connection to the server.

To be clear, we can launch any of our files directly from the Finder or with and OpenFile statement after a new generation, bypassing our initialization process, without any loss of server connection problems.

Our temporary workaround is to abandon the OpenSecret Initialize startup process.

Please let me know your thoughts. Much thanks.

I have two suggestions. Instead of opening the files as secret, try:

OpenFile “UnitMix”
MakeSecret

The .Initialize procedure will run because it opened fully. If you want to avoid seeing the UnitMix open, you could open it in a tiny window or offscreen. If you do see it open, you can verify that it connected as the wifi icon almost always appears when it is first opened in my experience. It disappears at will after that.

Secondly, and this kind of defeats the cool nature of the Pan X New Generation, you could only make new generations with the critical setting. That way, everyone will synchronize the new generation upon opening the databases again for the first time. This may be inconvenient but it may solve your problem until Jim can fix these issues.

Sorry, that should have been:

OpenFile “UnitMix”
MakeSecret

thanks for the reply. It is good to know that we are not the only ones experiencing this problem. This problem has been a real head scratcher. If I had a magic wand, the new generation would not require all users to shutdown. Instead users would experience a forced restart after the new generation is complete. Before a new generation all active sessions would be notified that data entry is going to be suspended while a new generation takes place requiring the clients to restart or refresh. As it stands I do not have a magic wand, and we are deeply appreciative of the amazing work Jim has done to get PanX to this point.

It was quite difficult to follow the sequence of steps you described (who’s on first?), but I think I have distilled it. Would you please review this bug report I have written up and verify that it matches what you are describing?

It is good to know that we are not the only ones experiencing this problem.

@Jeffk are you actually confirming that you have seen the problem @panaca is describing? Or are you just offering a suggestion (nothing wrong with that, just want to clarify).

That truly would take a magic wand. There is a very specific reason why all users are forced to shut down while a critical new generation is performed. If they weren’t, they might continue to modify the database while you work on the new generation. Whatever work they did during that period would be lost. It’s kind of like a race car having to pull into the pits to get the tires changed – you can’t change the tires while the car is moving!! Think of a new generation as a pit stop for your database.

When I first started using Pan X server, I was having problems verifying that databases opened secretly were actually connected. For now, I open all databases, then make them secret. It has eliminated that issue for me.

As you know, I have problems with the icons being displayed in the toolbar. I was offering suggestions to help mitigate the entering of data while not being connected to the server.

I conducted one test to see if I could duplicate the problem. I am not sure I understand the sequence, but here is what I did.
On computer 1, I opened database A, made some changes to a form, and edited a record. There were other users with connections to that database. I created a non-critical new generation of database A, then quit PanoramaX on computer 1.
On computer 2, I open database B, which automatically opened database 1 secretly. I can see in the server log that database A was opened by computer 2. (I did not see anything in the server log that the new generation was downloaded to computer 2; the log also did not confirm the a changed record was transferred. The log entry says “synchronized database: TimeClock (ts:14659-14659 changed:0 deleted:0)”
Finally, database A on computer correctly shows the changed form and correctly shows the revised data. So everything seems to have worked correctly, but I wonder about the server log; is it correct?

Jim, your description entered in bitbucket is pretty close. Here are some clarifications:

  1. Both non-critical and critical new generations made on a primary machine fail to update properly when a secondary machine opens the file via the OpenSecret statement. (Note the Preferences/Client on the secondary machine is set to update automatically.) To the observer, the database does not appear to be connected to the server (no wifi icon on a form after the database is made visible, and no wifi icon on the associated DataSheet.) When this happens, under the File menu, the user can see the option, Connect to Server, further indicating that the file is not connected to or in contact with the server.

  2. However, if the secondary machine quits Panorama after the case described in Paragraph 1 above and then opens the file again using the same code in the first attempt, the wifi icons appear on both form and DataSheet as expected. The option under the File menu is Disconnect from Server, if my memory is correct.

  3. A file that has not undergone a new generation on the primary machine can be opened by a secondary machine using the OpenSecret statement, and that file, when made visible, appears to be connected to the server on the first try. The wifi icons are displayed as expected.

  4. Also, we have observed that the Synchronize statement does not synchronize properly when a file is opened with OpenSecret subsequently followed by a Synchronize statement. We need to do some more testing on this, but that is another anomaly that we observed late today.

  5. We have replaced the OpenSecret statements with OpenFile statements, and that seems to solve the problem.