Offline EntitySync Functionality

[darrell73s]

Hi folks,

I'm exploring the use of the entity sync feature of the framework to perform some data replication. Because of my setup, I would need to use the "offline sync" portion of the functionality.

I have gone through the existing services, and the interface for offline entity sync in webtools. However, when it comes time to "Accept Offline Sync" or "Reject Offline Sync", a call is made to the updateOfflineEntitySync service which is not yet implemented.

Based on the existing flow, I would think that at the very least, an implemented updateOfflineEntitySync would update the runStatusId on the proper EntitySync/EntitySyncHistory record. Is it envisioned that there would be any additional logic (outside of what I've mentioned) in this service to handle the Accept/Reject of an offline sync operation?

If I decide to go down this route, I wanted to ensure that I understood the intended function of this service in case I wind up implementing something, and/or contributing an implementation for this service.

Thanks!
Darrell

[Jacques Le Roux]

This might help you figure out how entity sync works.

https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example

I have though never considered/needed what you call the "offline part"...

Jacques

Le 04/02/2015 23:35, darrell73s a écrit :

> Hi folks,
>
> I'm exploring the use of the entity sync feature of the framework to perform
> some data replication. Because of my setup, I would need to use the "offline
> sync" portion of the functionality.
>
> I have gone through the existing services, and the interface for offline
> entity sync in webtools. However, when it comes time to "Accept Offline
> Sync" or "Reject Offline Sync", a call is made to the
> updateOfflineEntitySync service which is not yet implemented.
>
> Based on the existing flow, I would think that at the very least, an
> implemented updateOfflineEntitySync would update the runStatusId on the
> proper EntitySync/EntitySyncHistory record. Is it envisioned that there
> would be any additional logic (outside of what I've mentioned) in this
> service to handle the Accept/Reject of an offline sync operation?
>
> If I decide to go down this route, I wanted to ensure that I understood the
> intended function of this service in case I wind up implementing something,
> and/or contributing an implementation for this service.
>
> Thanks!
> Darrell
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Offline-EntitySync-Functionality-tp4662586.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.
>

[darrell73s]

Thanks Jacques,

I had found that wiki article previously, which mainly documents the pull/push processes, and found the concepts helpful while going through the EntitySync source.

However, as you've indicated, what I'm looking at is the offline sync. Please let me know if the following would be better suited for the "dev" mailing list.

Assuming the scenario of an offline sync between the A (source) and B (target) nodes, here is how I believe it would occur between the two nodes (using the as-is WebTools flow in my example). Please feel free to correct my understanding, or if this scenario has already been hashed out, to point me in the right direction.

1) On Node A, user clicks "Run Offline Sync" which runs the 'runOfflineEntitySync' service. This service changes status on the EntitySync record, creates an EntitySyncHistory record, and produces an xml export of the data to be sync'd.

2) User on Node B inputs the xml file under "Load Offline Data" which ultimately runs 'loadOfflineSyncData' service. This service loads the exported data into the database by calling 'storeEntitySyncData'. There is a TODO in the code to "create a response document to send back to the initial sync machine".

3) Once there has been confirmation that the data was loaded into B, user presses "Accept Offline Sync" or "Reject Offline Sync" on Node A, which runs the 'updateOfflineEntitySync' service.

Questions/Thoughts based on the above:

* 'runOfflineEntitySync' service appears to be fully implemented

* 'loadOfflineSyncData' service appears to be almost fully implemented:
    - Service could be updated to produce the 'response document' mentioned in the TODO in cases where you may want to automate the third step in the process.
    - This service makes a call to 'storeEntitySyncData' which produces a series of stats related to records created/updated/deleted. Nothing is done with these stats such as putting them into EntitySyncHistory because no EntitySyncHistory is created by 'loadOfflineSyncData'.

* 'updateOfflineEntitySync' service is NOT implemented
       
1) In 'loadOfflineSyncData', would it make sense to call EntitySyncContext.runOfflineStartRunning() which would mark the status of the EntitySync record as ESR_RUNNING, as well as create the initial EntitySyncHistory record? Then, we could populate the stats returned by 'storeEntitySyncData' such that there is record of the offline load occurring on node B (target node), and its results into the EntitySyncHistory record we've just created, perhaps by running EntitySyncContext.saveResultsReportedFromDataStore()?

At this point in the process, we could also export a 'response document' with the results of the sync.

2) If 'updateOfflineEntitySync' were to be implemented, does it make sense for this service to: set the status of the EntitySync and corresponding EntitySyncHistory record based on whether: "Accept Offline Sync"/"Reject Offline Sync" button is pressed or optionally based on the response as sent in a 'response document'?

Thanks,
Darrell

[Jacques Le Roux]

Hi Darrel,

I have not used the EntitySync since fall 2009. Finally, I have maybe then used the "Offline EntitySync Functionality" but I can't clearly remember (I
remember I had issue with the automated EntitySync and had to get my hands dirty by indeed using Webtools and I remember it then helped).

On a trunk HEAD local instance, I have just tried to push the "Run Offline Sync" button on the 1st line (Id=1500) and got this error

2015-02-09 07:54:11,057 |OFBiz-JobQueue-1 |EntitySyncContext             |I| Creating EntitySyncContext with entitySyncId=1500
2015-02-09 07:54:11,124 |OFBiz-JobQueue-1 |EntitySyncContext             |I| In makeEntityModelToUseList for EntitySync with ID [1500] syncing 837
entities
2015-02-09 07:54:11,124 |OFBiz-JobQueue-1 |UtilMisc                      |I| null
java.lang.IllegalArgumentException: You must pass an even sized array to the toMap method (size = 3)
         at org.ofbiz.base.util.UtilMisc.toMap(UtilMisc.java:158) [ofbiz-base.jar:?]
         at org.ofbiz.entity.util.EntityQuery.where(EntityQuery.java:160) [ofbiz-entity.jar:?]
         at org.ofbiz.entityext.synchronization.EntitySyncContext.getCurrentRunStartTime(EntitySyncContext.java:894) [ofbiz-entityext.jar:?]
         at org.ofbiz.entityext.synchronization.EntitySyncContext.<init>(EntitySyncContext.java:188) [ofbiz-entityext.jar:?]
         at org.ofbiz.entityext.synchronization.EntitySyncServices.runOfflineEntitySync(EntitySyncServices.java:451) [ofbiz-entityext.jar:?]

Same with the 2nd line (5500), etc.

So clearly we need to revisit either OOTB data or code (seems more data since you seem to got it working, were you using the trunk?)

This said about your questions, please read inline

Le 07/02/2015 00:18, darrell73s a écrit :

> Thanks Jacques,
>
> I had found that wiki article previously, which mainly documents the
> pull/push processes, and found the concepts helpful while going through the
> EntitySync source.
>
> However, as you've indicated, what I'm looking at is the offline sync.
> Please let me know if the following would be better suited for the "dev"
> mailing list.
>
> Assuming the scenario of an offline sync between the A (source) and B
> (target) nodes, here is how I believe it would occur between the two nodes
> (using the as-is WebTools flow in my example). Please feel free to correct
> my understanding, or if this scenario has already been hashed out, to point
> me in the right direction.
>
> 1) On Node A, user clicks "Run Offline Sync" which runs the
> 'runOfflineEntitySync' service. This service changes status on the
> EntitySync record, creates an EntitySyncHistory record, and produces an xml
> export of the data to be sync'd.
>
> 2) User on Node B inputs the xml file under "Load Offline Data" which
> ultimately runs 'loadOfflineSyncData' service. This service loads the
> exported data into the database by calling 'storeEntitySyncData'. There is a
> TODO in the code to "create a response document to send back to the initial
> sync machine".
>
> 3) Once there has been confirmation that the data was loaded into B, user
> presses "Accept Offline Sync" or "Reject Offline Sync" on Node A, which runs
> the 'updateOfflineEntitySync' service.
>
> Questions/Thoughts based on the above:
>
> * 'runOfflineEntitySync' service appears to be fully implemented
>
> * 'loadOfflineSyncData' service appears to be almost fully implemented:
>      - Service could be updated to produce the 'response document' mentioned
> in the TODO in cases where you may want to automate the third step in the
> process.

That sounds good to me

>      - This service makes a call to 'storeEntitySyncData' which produces a
> series of stats related to records created/updated/deleted. Nothing is done
> with these stats such as putting them into EntitySyncHistory because no
> EntitySyncHistory is created by 'loadOfflineSyncData'.

 From the top of head I have no ideas about that

> * 'updateOfflineEntitySync' service is NOT implemented

It should be, but you see not much people use it, it's there waiting for more than 10 years.

> 1) In 'loadOfflineSyncData', would it make sense to call
> EntitySyncContext.runOfflineStartRunning() which would mark the status of
> the EntitySync record as ESR_RUNNING, as well as create the initial
> EntitySyncHistory record?

I would check using ESR_RUNNING there has no side effects on current code before doing this.

> Then, we could populate the stats returned by
> 'storeEntitySyncData' such that there is record of the offline load
> occurring on node B (target node), and its results into the
> EntitySyncHistory record we've just created, perhaps by running
> EntitySyncContext.saveResultsReportedFromDataStore()?

Same as above, we must be cautious about how theEntitySync mechanism works

> At this point in the process, we could also export a 'response document'
> with the results of the sync.

+1

> 2) If 'updateOfflineEntitySync' were to be implemented, does it make sense
> for this service to: set the status of the EntitySync and corresponding
> EntitySyncHistory record based on whether: "Accept Offline Sync"/"Reject
> Offline Sync" button is pressed or optionally based on the response as sent
> in a 'response document'?

This would need more review on my side. Since you are interested, I suggest you
1) create a Jira to fix the current issue with OOTB data, with a patch if possible. We could then use that for testing things
2) create Jiras with patches for the other features and improvements you suggest

Thanks!

Jacques

>
> Thanks,
> Darrell
>
>
>
> --
> View this message in context: http://ofbiz.135035.n4.nabble.com/Offline-EntitySync-Functionality-tp4662586p4662652.html
> Sent from the OFBiz - User mailing list archive at Nabble.com.
>