Reducing number of fields

OK, I checked and the paired, commented out select statements need to be replaced with execute statements to work. The other place I’d used calculated field names seems to work. So instead of my code above try this:

let donor="name of your donor db"
let big="name of your 859 db"
let first=date("10/1/1985")
for i,0,418
    window big
    ;if you want keep cells showing zero uncomment the first execute, if not uncomment the second
;execute |||Select not(emptycell(«|||+datepattern(monthmath(first,i),"mm/yy")+|||»))||
;execute |||Select not(emptycell(«|||+datepattern(monthmath(first,i),"mm/yy")+|||») or «|||+datepattern(monthmath(first,i),"mm/yy")+|||»=0)|||

    ifselect
    else
        window donor
        Field 3
        fieldname datepattern(monthmath(first,i),"mm/yy")
        importdatabase big,"existingdata","append","matchingfields","yes","selected","yes"
        if i=0
            select not(emptycell(unique))
            removeunselected
        endif
        Field Date
        emptyfill monthmath(first,i)
    endif
endloop
Field 3
fieldname "Amount"
Save

Be certain to have backup copies of the 859 db before trying this! If it completes check as much as needed to reassure yourself all the donations transferred ok. Once confident the next step is to delete the 419 fields from the 859 db. That can also be looped. Just how ‘contact information’ is left needing 440 fields you haven’t said, but at least the width problem would be halved then.

John,
Once again, thank you for taking all this time to help me get things sorted. It’s 4:12 here in Scotland and already dark, but I’ve still got work to do. I’m an American working here in the UK.
I had been able to get my big db down to ‘only’ 521 fields by eliminating a lot of dead wood, but, hopefully, with your help, I can eliminate 419 more fields. I have renamed my big db, big, just for simplicity sake in using your code. The donor db is donor again for simplicity. I can always rename them later, if I so choose.
I have a couple of questions regarding the code you sent me. I understand most of it, but there are still a few things I’m uncertain about.

  1. What do you mean by “for i,0,418”? Am I to replace the ‘i’ with either a zero or 418? If so, where do I put what, since there are several 'i’s in your code?
  2. You have an endloop statement, but I don’t see a loop statement. Is that OK?
  3. How will the proper donation be placed with the proper donor (the unique field)? If the donation doesn’t match up with the right donor, it is all for nought.
  4. Do I place your code in the ‘code’ box in the field properties for the “Amount” field? Will it run automatically, or do I need to do something for it to run?
    Again, thanks for all the help! I really think this will work and be much more efficient.

“For” is a kind of loop, new to Panorama in 10.1, but ancient in other computer languages. So the endloop is for it. Look it up in Help. ‘i’ is the variable that holds its counter and is used as such in several places. It could be called anything but Help used “i” and I kept that. So the first time through the loop it adds i=0 months to the “first” variable’s staring date and adds one more month until in the last pass you are 418 months later.

By the magic of the “matchingfields” option of importdatabase. Code has the loop rename what will finally be named the “Amount” field in Donors sequentially to the name of your monthly fields and having the first field in Donors named identically to the unique field in the 859 db (and not having a field named Date in 859 db). So each time you import only data in those two fields with matching names get imported. Because importdatabase is using the “append” option for “existingdata” you keep adding months of data to it, just combined in one field. Each imported record carries its unique field data along.

Put this code in a new procedure window. You can use the Run button there to use it and once transferred use the Remove button to delete it. This is separate from whatever code you’ll write so the individual databases work together properly, along with any forms you design. This is just get your old data into a more serviceable form. Once you’re confident that data transferred ok I can give you a simpler loop to delete the then no longer needed 419 fields from the 859 db. Which should make whatever further you want to improve the organization of the mailing list etc left there more responsive. Check out Help on the Morph field dialog for some ideas on that. Once you’ve figured how you want data organized in your new PanX databases and gotten your legacy data into that form then you can figure out how to simplify and automate future actions.

John,
That was exactly what I needed to know. As it is ‘tea time’ now, I will run it tomorrow and let you know what happened. I am getting excited about the new possibilities and hope to explore PanX a wee bit more with regard to my bibliographic db which I use far more, once I get this donor db sorted. Thanks again.

If Will can get his hands on the Panorama 10.2 early release he will be able to link the files much more easily using the new relational tools that it provides. Makes using a boatload of lookups history. :partying_face:

2 posts were merged into an existing topic: BIG NEWS – Panorama X 10.2 Classes & Preview starts in January!

Downside of jumping to 10.2 is its relational tools are new concepts to some, including me. Feels to me that Will could use more time to absorb how PanX does things that are conceptionally similar in his familiar Pan6, before moving onto new concepts.

I’ve got a boatload of lookups, but I also have a boatload of experience working with them, knitting complex long procedures across numerous databases. Which currently work! Just not as quickly as I could wish. I have no experience working with the kind of databases who’s users have historically sneered down on Panorama as “not-relational.”

Jumping too fast into the new risks breaking the old. The logical solution is the coming course. Amongst his other virtues Jim is a good teacher. I’ll probably watch the videos after they’re recorded. The prior videos worked well for me. Got my recoded PanX conversions to run about as fast as my kludgy prior Pan6 programs! Then try them in small ways where they can’t hurt much, until I master their concepts. Then gradually blend them into my old code as time permits. Then probably dream up some now features to add.

John,
On my first attempt to run the code you had sent me, I realised that I needed to change my Language and region system preferences, since the way dates are configured here is different than in the US (here it is dd/mm/yy). When I tried again, I got the following error message: “Field or Variable [else] does not exist.” That is referring to the following lines in your code:
execute |||Select not(emptycell(«|||+datepattern(monthmath(first,i),“mm/yy”)+|||») or «|||+datepattern(monthmath(first,i),“mm/yy”)+|||»=0)|||

ifselect
else
    window donor
    Field 3

How do I get your code to work?
Thanks.

I haven’t been following this closely, but the lines you referred to looked strange to me, so I checked ifselect in help. “ifselect” starts a block of code that is terminated by either else or endif. There’s no code here between ifselect and else, so it seems as if Pan is trying to interpret else as part of that block of code. Shouldn’t it be

Ifselect

…[some instructions]

else

window donor

Field 3

Etc.

Yes, I’d goofed with the ifselect. I dream I can code the bullseye, but am satisfied if I can find it incrementally from what I first write. To do what I’d intended change it to:
if info("empty")

Keep the else and the rest.

Your region preferences shouldn’t affect this. Dates are stored as the same integers in all regions. Those preferences just affect their default format as text. In this that’s all pre-empted by the datepattern() functions

John,
You are way beyond me.
I changed the code as you instructed. However, I then received the following error message:
sizeof( function error: no field or variable named “1/85”.
What should I do now? We’re getting there, slowly but surely and I certainly very much appreciate your patience and help with this.

I spoke to quickly on the region preferences. The date series was supposed on Oct 1985, ie. 10/85. But the error is saying no field named “1/85” When I wrote:
let first=date("10/1/1985")
I was thinking US style, Oct-1-1985. With your region setting it now meant Jan-10-1985 so is starting to transfer 419 consecutive months with “1/85” Change that to
let first=date("1/10/1985")

John,
It worked!!! I’ll have to check to make sure all the data which transferred is correct, but I know how I can do that, without too much difficulty. THANK YOU!!!
Could you send me that loop code to remove the 419 fields from my big db? Then I can just tweak the two dbs and I’ll be good to go, thanks to you. Thanks again.

Make sure you have adequate backups, but this will remove the 419 fields. Put it in a new procedure window of the 859 db (donor db doesn’t need to be running for this one) and run it there.

let first=date("1/10/1985")
for i,0,418
    execute |||field «|||+datepattern(monthmath(first,i),"mm/yy")+|||»|||
    deletefield
endloop
save

For future reference, you can use the datevalue( function to specify date constants in a manner that is agnostic to system settings, so October 1, 1985 can be generated using this formula:

datevalue(1985,10,1)

The formula above will work correctly no matter what your system settings are.

There are three design areas that seemed to come up more often than others when I have had to “fix” database files.

First - and maybe ProVue is a teensy bit responsible - is the user is thinking horizontally (fields) instead of vertically (records). I’m guessing that’s because their introduction to data was a spreadsheet; like a pro-forma statement. They’d see months or years as column labels across the top and various categories of profit and cost for those months down the page. And ProVue would say something like, “The database that thinks it’s a spreadsheet”. So people would use that structure to hold their data - each day, week, month, or year would become a new field extending horizontally across the data sheet.

Instead, the categories should have been the fields with one field holding the time interval (day, week, month, or year).

As data is entered, you should see your database growing in records, not in fields.

The second area was duplicated data. I once had to correct a business address and the user wondered why it took so long. In that database, the address was entered in at least three different places - and I had to go through more than 20 procedures in each of at least five different files to find them all. And they were entered in three different ways - hard coded into a procedure, live entered into a field, and displayed via a flash art object. How much easier to have that data in one place - even if in its own database with one record - and just link to it when needed. If it needed to be changed, it only had to be changed in one place.

Another place this shows up is form is Duplicated (cue Gary Larson starting another “Trouble Brew’n” series. One form has editable fields for data entry, the second form has fixed fields for display. Time passes - the user needs a change on the form Just something little - how hard can it be. Except it needs to be change not on one form, but two.

As a general rule, try not to duplicate.

The third area is not seeing the usefulness of an intermediate file. A standard example is the classical - I want to see all the classes this student has taken or I want to see all the students who have taken this class. Usually I’ve see this attempted by creating a student record and adding a record that copies over class information. Hopefully the user avoid the first issue and adds a record for the student with class info instead of adding a new class field.

The same thing is happening in the class database - a new record (instead of a new field) is added for each student. But this often falls into the second issue as class info is copied into the student record, causing some data duplication.

One solution is to create an intermediate database that just has the student ID, Class ID, and Date. If the student takes a new class the student record (profile) doesn’t change - a new record is just added to the intermediate file. Now if you want to know all the classes a student took, you can select all the records for that student in the intermediate file and display (lookup) all the class names (and other data) via the Class ID from those selected records. Works exactly the other way - all the students who took a class - select the intermediate records for that Class ID, and display the student by the student ID in those selected records.

The above is students and classes - but the same structure comes up again and again. For example, “What products has this customer ordered?” and “What customers have ordered this product?”

A “half-issue” might be considering the difference between history vs current time. The user wants the invoice database to pull over product information either when the invoice is displayed or a product ID field is exited. But if a product price changes and you are looking at an old invoice, you don’t want those fields to update to the new product prices. You want it to stay a historical record of what happened at that time. And that can get tricky because if you say, “Don’t change anything if some field already has something in it.”, then you’ve sort of locked it. If you don’t want to have to clear the field first, perhaps you can modify your “lock” to allow a change if, say, the Option key is down on exit.

But mainly try to avoid growing the database horizontally, entering the same data in multiple places (that goes for duplicating procedures instead of linking to one), and consider the advantage of having an intermediate “linking” database to bring associations together. Again - not “hard and fast”, no exceptions. More like considerations.

Now this is all “old style” technique/technology and PanX - especially PanX 10.2 and the new server - will probably have alternate/better/sutomatic solutions to some of those issue. I was probably influenced too much by the more mature Twilight Zone - Outer Limits. The shows tag line was, “… We control the horizontal. We control the Vertical …” sometimes automatic is good. Sometimes it’s good to be in control.

John,
Great. Thank you for this. I do have several back ups, so I should be safe. Thanks again for all your help.

Yes, this was my ‘problem’ way back when I initially designed my db. I thought it was OK to keep adding fields and that I needed to be careful of how many records I added. Now I understand that it’s the other way around.
I will be looking forward to PanX.2 and the relational tools it will contain.

Gary,
I think John is right. I will need to get more familiar with PaxX before I jump right in to PanX.2. I’m not trying to push the envelope with the things I’m doing with Panorama.

We stopped using that marketing message over 25 years ago!!! We stopped using it precisely because it caused misleading expectations. Though obviously it was a message that resonated with people – they still remember it all these years later.