Excessive Submit Processing with 40K+ Categories

[Phil]

Since I can't run my 40K+ state, county, city and zip code index on the standard version, I thought I'd try Version 2. Unfortunately I'm having and similar, but different problem. Whereas Advanced Search takes many minutes and uses about as much memory as I can give it on my 2Gig development server, Version 2 on the other-hand does similarly when attempting to do a submit:

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
11910 www-data 20 0 361m 28m 4552 R 96 1.4 1:29.24 apache2

And this is what I get when I let it go:

Fatal error: Maximum execution time of 360 seconds exceeded in /var/www/phpld/include/functions.php on line -555631244

I can write a simple submit that gets the job done by just doing the appropriate table updates, but before doing so I thought I'd ask whether there might be some nice but unnecessary code that's using all the resources that I could deactivate. Any thoughts would be appreciated.

[James]

this is with ajax enabled? in the 3.x version? and iwth the advanced search what version of the script?

[Phil]

James,

You can refer to our forum exchanges at:

http://www.phplinkdirectory.com/forum/showthread.php?t=22540

regarding your question about which version (version 3.3) I referenced.

I host my 60+ sites with either hostgator.com or bluehost.com. It was bluehost.com where I tried to run version 3.3 and where I finally deactivated advance search because so many folks where trying it to see it fail.

I don't think it makes sense to install either version on either of my hosting accounts given both my foreknowledge that they will both timeout.

I just wanted to know if anyone had any idea what processing was going on that was consuming so much time. Recall that the results finally come back after upwards of 20+ minutes of wall time running at max memory and cpu on a 64 bit dual core server when running advanced search on version 3.3.

I get along with PHP, but not Smarty where I think the activity is taking place, so I just thought I'd ask.

==============

As an FYI, I just did my quasi submit by updates through MySQL, which took just split seconds as expected and the link works just fine. If you nor anyone else have any thoughts within the next few days, I'll just write my own submit 'cause once the data get's in the database it works great...

[James]

advanced in 3.4 should work

[Phil]

James,

I just tried version 3.4 with the same results as I was experiencing with the earlier version. In addition, I tried doing a submit with version 3.4 and got the same excessive usage as with version 2.2.

I think it boils down to the processing that is done to recreate the drop-down directory structure for both doing submits as well as Advance Search.

Other directories with which I have experience require the user to browse down to the category into which they which to submit a link and are thus able to do the instant simple insert I mentioned earlier.

So, unless you've got some other thoughts, let's forget this request given that your current architecture works very well for smaller directories.

Thanks.:)

Phil

[James]

pm me the url please and you can do a fixed select for the submit page

[Phil]

James,

I've installed version 3.4 on http://www.hostpdq.com, but will be happy to switch it to 2.2 if you like.

Phil

[David]

I clicking through some pages just now in two different browsers. I found that 9 out of 10 pages loaded quickly, but then there would be an occasional page that took 3-4 seconds. If the there is variability, that is usually a sign of something going on with the server. Generally, if there is a script problem, the slowness will be consistent, and I am seeing lots of pages load for the first time in the browser very quickly.

As James pointed out, turning on AJAX category selection will eliminate the problem with load time of categories. What it does it make it so only one level of the category structure is loaded at a time. Once you hit more than a couple thousand using the drop down method the page is just too large.

[Phil]

David,

Using AJAX makes no discernible difference:

Category selection method: Regular Selection (Default) Defined (no selection possible) Live category selection with AJAX Type of category selection to use on submit forms. For more than 5000 categories, we strongly recommend the "AJAX method"

CPU is max'd out on my standalone development server. However, since 5000 is only about an 1/8" of the initial set of categories, I can only wonder how I would go about accessing remaining categories if this option stops without some means of continuing onward to remaining categories.

As I mentioned earlier, I think the only realistic way in which a large number of categories can be processed speedily is to browse down to the appropriate directory as the means of selection rather than building a large drop-down menu for each use, which is likely to be substantial.

I'm not asking that you make such a significant alternation in your architecture to use this more conventional approach, but rather just explaining the results of using AJAX and my opinion.

Given that I've got 40+ domains waiting for the ability to make use of a large geographical directory, I've been working on my own directory off and own, so I'll be good to go sooner or later anyway.

I don't know how many other domains my ISP has on the server I'm using, but from what I can see in the log files, there's only one. Either way, when I can ssh in and have even the biggest installs install with no discernible hesitation seems about as good as I could expect. I don't see the browser hesitation you speak of from here. I did notice yesterday that your server was timing out for several minutes so perhaps the hesitation is at your end.

Anyway, thanks for your support...

Phil:)

[James]

You say dev server.... where is it located and who set it up?

I have worked on one at the planet for someone here that it was just a bad mysql config.

Email me the category dump please to the paypal email in my sig and I will toss it up on my server for load testing.

Without actually working on an install with the number of categories then it is a waste of time.

[Phil]

James,

Sorry about the delay in getting back with you; I've been busy to say the least.

Now, insofar as my development server:

Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.10) Gecko/2009042513 Ubuntu/8.04 (hardy) Firefox/3.0.10 always kept up to date with the latest of all software releases with 4 Gigs of memory.

I set it up on my premises. I do not run email; ssh; ftp; etc., and it has no external access from the outside world. That is; I and I alone set it up and I and I alone ever use it.

When I have an application setup and operational I install it on one of my Bluehost.com or HostGator.com hosting accounts. I never, knowingly install software on any of my accounts that would in any way compromise my relationship with any of my hosting accounts; consequently, I will not install my Category / Location directory for testing now that I know the results of so doing. You'll perhaps recall I did several months ago and each time anyone attempted to do an advanced search, it stopped all of the hosting accounts dead in their tracks that were running on that particular Bluehost.com web server; mine and many others.

Now, insofar as data with which to test; you can check with David, to whom I sent a download of these State; County; City and Zip categories. If he doesn't have it any longer, I will set about creating another - just tell me whether you want to use the latest free or latest paid version.

From my testing it appears that the problem for both versions is the time it takes for either version to build a dropdown box containing all of the categories from which to select. I have opened up my development server so that it doesn't time out and after a little over 20 minutes of wall time and with both CPU's max'ed out at or near 100% utilization, the drop down box is created and selectable. Consequently, it would appear, to me, that the application logic is working as designed.

The only directories that work flawlessly and instantly with these categories are those that require the user to traverse the directory structure down to where they want to add their advertisement. So I have pretty much given up on using phpld for any directory in which I need a large set of categories; that is, one in which users wish to submit local advertisements.

Although it's pretty much hit or miss, I am building a directory specific to my needs; which though nowhere near as robust as phpld, has no problem with large directories, so while I would prefer to use phpld, the urgency of my needs have subsided, hence you can ignore any subsequent concerns regarding this issue; unless of course you provide a browse down option rather than the current dropdown box option.

Phil

[James]

I really need the cats to test
If i understand you do not want to use ajax but want to use the dropdown?

3.4 version and pm me the download.