Usersync for a large user table vB testbed keeps failing

[Tigratrus]

Please fill up the following form to enjoy free support from our Team. No Support will be provided in case this form is not filled up!

1. Have you searched the forums and checked the documentation section for a solution?
I've run numerous searches for usersync errors but haven't found anything that specifically addresses usersyncs with large user dbs, though I'm sure they are there.

2. Please summarize your problem in a few lines.

I'm trying to work out the bugs in rolling out jFusion on our test server before setting it up on the live system. The test server has a user table of about 85,000 users, the live site is about 95,000. On the test server the usersync dies every time. I upped the wait_timeout on the my.cnf to 1800 (which should be 1/2 hour)and I was able to get it to get almost down to 70,000 left to sync before it died. When I run it on normal detail it just stops progressing, when I run it on extended it errors out with a "This webpage is not available." after a looong wait. In neither case is it anywhere near the 1/2 hour wait_timeout (which seemed to be causing a problem before I increased it).
Is there a way to reduce the time delay between cycles? If it was just waiting a bit over 1 second instead of 10, it might be able to get though it if there's a timeout coming into play.

3. What steps will reproduce the problem?
Try to run a usersync

4. What is the expected output? What do you see instead?
Usersynch fails. And it always restarts at the beginning so it can never complete the process.
5. You must include a printout of the login checker in the JFusion component.

Code: Select all

Server Configuration
Joomla Version   1.5.8
PHP Version   5.2.5
MySQL Version   5.0.67
System Information   Linux vps.ikeafans.info 2.6.18-028stab060.8 #1 SMP Mon Feb 9 20:25:36 MSK 2009 i686
Browser Information   Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.19 (KHTML, like Gecko) Chrome/1.0.154.65 Safari/525.19
JFusion User and Auth Plugins   Enabled

JFusion Versions
JFusion Component Version    1.1.2-001
JFusion Authentication Plugin Version    1.1.2-001
JFusion User Plugin Version    1.1.2-001
JFusion Activity Module Version    1.1.2-001
JFusion Login Module Version    1.1.2-001
JFusion Search Plugin Version    1.1.2-001
JFusion Discussion Plugin Version    1.1.2-001

JFusion joomla_int Plugin
configuration   master 1
slave 0
dual_login 1
check_encryption 1
JFusion vbulletin Plugin
configuration   master 0
slave 1
dual_login 1
check_encryption 1


Authentication Plugin
userid   73
activation   ""
username   Tigratrus
name   Tigratrus
password   8a8c7b********
email   james@ikeafans.com
block   0
group_name   Super Administrator
group_id   25
params   admin_language= language= editor= helpsite= timezone=-5
password_salt   N2be*****
   Authentication Plugin Success


jfusion User Plugin
   User Plugin Success
User Plugin Debug
init   0 Using userinfo JFusion authentication plugin
vbulletin User Update   0 A user has been found that matches the login details
1 Skipped Password Update:Password already valid
vbulletin User Details   userid 2339
username Tigratrus
name Tigratrus
email **deleted**
group_id 6
group_name Administrators
membergroupids 60
displaygroupid 6
password **deleted**
password_**deleted**
usertitle Administrator
customtitle 1
posts 2654
block 0
activation ""
password_clear ******
vbulletin Session   Created session: userid = 2339, password = **deleted**, Cookie Salt = **deleted**, Cookie Path = /, Cookie Domain = www.ikeapedia.com


Thanks in advance for any assistance you can provide. I know Nathan was able to get a large vB forum to sync, so it's possible, but I've been banging my head against the wall for quite a while trying to make it work.

I'm wondering if it's the:
connect_timeout=120
interactive_timeout=300

that might be breaking it now?

I'll try setting those both to 1800 as well and see how that goes...

[Tigratrus]

By bumping all those values up I'm able to get the usersync to run for about 5 1/2 minutes before it stops making progress and I have to click the "Click here if your usersync does not progress anymore" link. Problem is it starts from the beginning each time, and while it moves through the accounts that have already been done a lot faster, as I get farther though the sync it eats up more and more of my 5.5 minutes going though the already synced accounts. I don't think it's going to be possible to get this to finish... Not to mention the fact that it's taking hours to move though the user table in ever smaller chunks.

Can we decrease the 10 second wait time or increase the # of records it processes on a pass? Or both? Please?

Thanks in advance for any assistance anyone can provide!

James and Susan

[mariusvr]

Hello old friend Great to have you back......

What does the stats in the usersync history say? (can you copy and paste those details here). The sync should resume for where it left off, it would be great to find out the issue, as 70.000 users is probably one of the largest sites running JFusion.

Thanks, Marius

[Tigratrus]

Hi Marius!

::waves::

Result 1: "Normal Output"
With it set for normal output the sync stalls in about 5.5 minutes and you have to click the link as said above. When I go in and look at the usersync history I see:
Total time: "Usersync did not finish:
Details:

Code: Select all

Your usersync has finished, because there was nothing todo. (No plugins were selected for synchronisation)

Even though there WAS a plugin selected. I know there was a plugin selected, because (a) I've been very meticulous , *and* (b) because if there isn't the usersync completes immediately with that message, it doesn't run for 5 minutes 20 seconds and eventually stall out;).

The other interesting thing is that it starts back at the very beginning of the sync everytime. Until it gets to where it stalled last time it goes though them about 1000-1500 records at a pass, then it drops to about 50-100 when it gets to the point that it's processing new records.

Result 2: "Extended Output"
When run in "Extended Output" mode, the sync fails with a "This webpage is not available." browser page error with the URL of: "http://www.ikeapedia.com/administrator/index.php?option=com_jfusion&task=syncstatus&syncid=UbPkSVlMZg&action=master&slave[vbulletin][perform_sync]=1"

When I go in and look at the usersync history it's:

Code: Select all

Usersync Status:
Total Time: "Usersync did not finish"
Plugin Name    Total users awaiting sync    Users Unchanged    Users Updated    Users Created    Users Deleted    User Conflicts
vbulletin   40827   39845   3502   559   0   402

James and Susan

[Tigratrus]

Just updated to 1.1.3 RC and the Extended output is ending right on five minutes in, and always with the same:
72947 members awaiting sync remaining. It never gets any farther than that.

So far Normal output is running though about 400 of the already processed members per pass three minutes into the sync so I expect it to stall in about another two.

Yup... Stalled out partway through the pass that crossed over the five minute mark. So something's breaking right at 5 minutes. No DB error since I increased the three MySQL timeout values to 1800.

Is there any way to increase the # processed per pass or reduce the 10 second wait time? I seem to be up against a proverbial brick wall here .

James and Susan

[mariusvr]

With the extended output mode, the sync fails because of time-out issues.

With the default usersync mode (that uses ajax to prevent time-outs), I believe that the php memory limits is being exceeded, which results in the usersync stopping. With a large userset, the user conflicts are being saved for later retrieval and i believe that this conflict variable can cause issues if 10.000's users are being synchronised. The solution will be to change the way the user conflict data is stored. Rather than keeping all conflicts in memory, it will need to be unset so that no conflicts are stored in the memory itself (only in the usersync file in the jfusion admin folder). You could also try to increase the memory limit, but that will most likely fail as well.

I will put it on my todo list, but it will most likely be 2 weeks before a new release of JFusion can be created (we are working on even more revolutionary new features0

Thanks, Marius

[Tigratrus]

Marius,

Just checked, memory_limit is set to 132M. I can certainly increase that if it would help... Our server has 17.2G memory available for burst use, 8G guaranteed(we usually use 400-800M).

The thing that's puzzling is that I'm not getting a php error when the sync dies? Which I would sort of expect if it was the memory limit.

Is there a way we could up the cycle time or number processed in each pass?

We've got a major new system that we're almost done with that would work a *lot* better if we could shift to using J! as the accounts master, but based on my testing I don't think that will work if the users aren't sync'd up to start?

James and Susan

[Tigratrus]

Nope. That's NOT what's causing it to stop...

Upped the memory_limit to 1000M
max_execution_time to 240
max_input_time to 120

in php.ini and it's still failing *exactly* where it was before, right as you cross over the 5 minute threshold.

James and Susan

[Tigratrus]

Ah HA!

Sometime you just need to do some brute force searching...

Went looking in the J! PHP Information page for "300" as 5 minutes * 60 seconds = 300.
Guess what I found?
Apache settings:

Timeouts Connection: 300 - Keep-Alive: 300

Coincidence?!

went into httpd.conf and changed them from 300 to 1200 and restarted apache.

BANG! It gets past the 5 minute mark!

I'll probably have to up it a lot more to get it to complete, but that certainly seems to be the limitation we're hitting.

Follow-up:

After letting the sync run (the one that got past 5 minutes above) I had my stopwatch running and watched it like a hawk as it hit 20 minutes. Guess what? The sync stalled *right* as it crossed 20 minutes aka 1200 seconds.

So, that seems pretty definitive to me. I think I'll probably need to bump it up to 3000 to get it to finish the sync...

James and Susan

[Tigratrus]

'K.

So what we've learned here is that Apache timeouts *are* going to be a major problem for large usersyncs. The Usersync history isn't reporting it as a timeout, it's saying

Code: Select all

Your usersync has finished, because there was nothing todo. (No plugins were selected for synchronisation)

so future admins that hit this will be clueless as to what's happening.

Otherwise I'm not sure how to handle the time outs... Is there a way to spawn a new process each AJAX update instead of continuing the original one? Something so that apache doesn't see it as a process that's locked into an infinite loop?

James and Susan