Usersync for a large user table vB testbed keeps failing

[digitaldoctor]

added
http://www.jfusion.org/index.php/docs/doku.php?id=installation;user_sync

[mariusvr]

Would you be able to email or pM me a SQL extract from your jos_jfusion_sync table?

I will then be able to improve the error message, to say that a time-out occurred and show a resume button (rather than forcing users to start their sync over again).

Thanks, Marius

[Tigratrus]

Marius,

Sent you a PM .

You might want to take a look at how vBulletin handles this sort of batch job? Instead of having it one long continuous process, You provide a # of records to process at a time. The process kicks off, processes that many records then marks where it got to in the tables and stops. After the time interval passes, an AJAX event starts the process again, and it picks up where it left off and processes the next chunk, etc. So you're never trying to run a 4 hour long process straight through, it's run in small chunks one at a time and there is never a possibility of a timeout or memory limit getting hit.

My math suggests that our real sync on the live server is going to take between 4 and 6 hours, that's a REALLY long process to try and run non-stop, and not everyone is going to have the luxury of being able to go in and set an absurdly long timeout value on the apache to let it run. Just some feedback on behalf of future admins .

In case I haven't mentioned it yet, I did want to stress how nice it is to see the dual login working flawlessly! Major congrats to all involved! And having the vb userid in the jos_jfusion_users_plugin table is fantastic, it'll make some of the things we're trying to do a *lot* easier . JFusion is looking great!

James and Susan

[Tigratrus]

added
http://www.jfusion.org/index.php/docs/doku.php?id=installation;user_sync

Thx Doc!

That's the main reason I've been posting so much on this thread, I figure lessons shared are ones someone else can find if they have problems .

James and Susan

[Tigratrus]

Marius,

Did you ever get the PM? I couldn't attach the file you wanted here or in a PM. Do you want me to email it to you? If so what address?

Alan et al,

Has anyone looked at the idea of making the user sync process in chunks rather than one looooooooooooooong process? Setting up a server to allow 5 hour+ apache time outs is NOT something that many people going to be able to do... If you changed the sync processing so that it processed like 1000 users per process (ideally let the admin specify how many per update), marked it in the DB then ended the process and just keep kicking off a new process (via AJAX?) until they are all synced, that would totally eliminate this as an issue.

I know there is existing code in vB for this sort of thing that could probably used as a reference? Not to be directly reused, as it obviously needs to be non-forum/software specific, but it shows how they kick off a series of small self contained updates to work with large numbers of records in batches and avoid any timeout issues.

James and Susan

[mariusvr]

Yes an email would be great: mariusvr at gmail.com

The usersync already stores in jos_jfusion_sync table where it is up to in terms of usersync. Then is the sync times-out, the resume button will allow you to pick up where you left. Your issue is that the 'resume button' does not show up in the usersync history screen.

Thanks, Marius

[Tigratrus]

Marius,

Thanks... You should have a copy of the sql dump from the jos_jfusion_sync table in your email . Just let us know if there's anything else we can do to help!

James and Susan

[Tigratrus]

Is there something we can do as a workaround? I'd really like to shift to using J! as the accounts master, but given the slow down I've seen on our server when I set the apache timeouts to 5+ hours I don't know that I'll be able to make it all the way through the sync... It would REALLY be bad to get through 4 hours of it and have the server lockup.

If it's just a question of making the resume button show up, is there another way to trigger a resume? Does the button link to a specific URL that we could simply put in manually?

James and Susan