Entity Sync
Locked
 
|
We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another.
Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a collection of entities to sync (pull or push) between delegators (or service). In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting point. Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and their contact info, notes, preferences, etc.... Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, products, and subset of pricing, product content, features, etc.... to other instances. Looks like a modification to entity sync is required to do this? or am i mistaken. Modification involves: - build set of entities in sync group - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, what they have done. Marc Morin Emforium Group Inc. ALL-IN Software™ 519-772-6824 ext 201 [hidden email] |
Administrator
|
Hi Marc,
Did you follow recent threads on this subject ? Jacques From: "Marc Morin" <[hidden email]> > We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. > > Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a > collection of entities to sync (pull or push) between delegators (or service). > > In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting > point. > > Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and > their contact info, notes, preferences, etc.... > > Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, > products, and subset of pricing, product content, features, etc.... to other instances. > > Looks like a modification to entity sync is required to do this? or am i mistaken. > > Modification involves: > > - build set of entities in sync group > - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') > - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. > - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. > > > I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, > what they have done. > > Marc Morin > Emforium Group Inc. > ALL-IN Software™ > 519-772-6824 ext 201 > [hidden email] > > > |
|
|
Rookie error... apologies
----- Original Message ----- From: "Jacques Le Roux" <[hidden email]> To: [hidden email] Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern Subject: Re: Entity Sync Hi Marc, Did you follow recent threads on this subject ? Jacques From: "Marc Morin" <[hidden email]> > We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. > > Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a > collection of entities to sync (pull or push) between delegators (or service). > > In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting > point. > > Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and > their contact info, notes, preferences, etc.... > > Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, > products, and subset of pricing, product content, features, etc.... to other instances. > > Looks like a modification to entity sync is required to do this? or am i mistaken. > > Modification involves: > > - build set of entities in sync group > - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') > - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. > - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. > > > I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, > what they have done. > > Marc Morin > Emforium Group Inc. > ALL-IN Software™ > 519-772-6824 ext 201 > [hidden email] > > > |
|
Administrator
|
Do you mean it was a simple error on your side (sorry I did not take the time to read all yet ;o) ?
Jacques From: "Marc Morin" <[hidden email]> > Rookie error... apologies > > ----- Original Message ----- > From: "Jacques Le Roux" <[hidden email]> > To: [hidden email] > Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern > Subject: Re: Entity Sync > > Hi Marc, > > Did you follow recent threads on this subject ? > > Jacques > > From: "Marc Morin" <[hidden email]> >> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >> >> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >> collection of entities to sync (pull or push) between delegators (or service). >> >> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting >> point. >> >> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and >> their contact info, notes, preferences, etc.... >> >> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >> products, and subset of pricing, product content, features, etc.... to other instances. >> >> Looks like a modification to entity sync is required to do this? or am i mistaken. >> >> Modification involves: >> >> - build set of entities in sync group >> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >> >> >> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, >> what they have done. >> >> Marc Morin >> Emforium Group Inc. >> ALL-IN Software™ >> 519-772-6824 ext 201 >> [hidden email] >> >> >> > > > |
|
|
In reply to this post by Marc Morin
Marc, Actually this is a separate topic from what I've seen in the other thread. It would be good to read the other thread, but I'd recommend taking everything there with a grain of salt. There seems to be some misunderstanding in a lot of the assessments of what the Entity Sync does and what the issues are with it (ie things that make it slow, the constraints it has trying to reproduce time order of database changes over multiple tables, etc). In any case, yes, it is correct that the current Entity Sync stuff does not have filtering on a record level (other than by time), and also has nothing to follow a graph of related records to include desired information. -David On Oct 27, 2009, at 8:24 AM, Marc Morin wrote: > Rookie error... apologies > > ----- Original Message ----- > From: "Jacques Le Roux" <[hidden email]> > To: [hidden email] > Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada > Eastern > Subject: Re: Entity Sync > > Hi Marc, > > Did you follow recent threads on this subject ? > > Jacques > > From: "Marc Morin" <[hidden email]> >> We have a couple of use cases where we need to sync a subset of the >> information in one database and mirror it into another. >> >> Been looking at the entity sync capability that is present in >> ofbiz. Very good start, ability to build a sync set that is a >> collection of entities to sync (pull or push) between delegators >> (or service). >> >> In my case, I need to create the nest of entities to consider for >> the sync, but I need to restrict the rows from some starting >> point. >> >> Use case 1: Want to import a collection of parties from one system >> along with their relationships between the set of parties and >> their contact info, notes, preferences, etc.... >> >> Use case 2: Create a product category, rollups, and then associate >> products to that category. Want to publish this category, >> products, and subset of pricing, product content, features, etc.... >> to other instances. >> >> Looks like a modification to entity sync is required to do this? or >> am i mistaken. >> >> Modification involves: >> >> - build set of entities in sync group >> - add condition restrictions on possibly any number of entities for >> candidate rows. (category_id = 'x') >> - "soft" inclusion by foreign key reference from a row in the set >> to one that is not explicit. >> - alter row for foreign key references outside of entity set to >> null or to rows that cannot be replicated. >> >> >> I'm sure more precision is needed here and will to do the work. >> Just wanted to see if others have similar use cases and if so, >> what they have done. >> >> Marc Morin >> Emforium Group Inc. >> ALL-IN Software™ >> 519-772-6824 ext 201 >> [hidden email] >> >> >> > > |
|
Administrator
|
Marc David,
Sounds like David took the time to read you message... I remains, that we need to clarify what is working and what is not. I should work on this these next days. Jacques From: "David E Jones" <[hidden email]> > > Marc, > > Actually this is a separate topic from what I've seen in the other thread. It would be good to read the other thread, but I'd > recommend taking everything there with a grain of salt. There seems to be some misunderstanding in a lot of the assessments of > what the Entity Sync does and what the issues are with it (ie things that make it slow, the constraints it has trying to > reproduce time order of database changes over multiple tables, etc). > > In any case, yes, it is correct that the current Entity Sync stuff does not have filtering on a record level (other than by > time), and also has nothing to follow a graph of related records to include desired information. > > -David > > > On Oct 27, 2009, at 8:24 AM, Marc Morin wrote: > >> Rookie error... apologies >> >> ----- Original Message ----- >> From: "Jacques Le Roux" <[hidden email]> >> To: [hidden email] >> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >> Subject: Re: Entity Sync >> >> Hi Marc, >> >> Did you follow recent threads on this subject ? >> >> Jacques >> >> From: "Marc Morin" <[hidden email]> >>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>> >>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>> collection of entities to sync (pull or push) between delegators (or service). >>> >>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting >>> point. >>> >>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties >>> and >>> their contact info, notes, preferences, etc.... >>> >>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >>> products, and subset of pricing, product content, features, etc.... to other instances. >>> >>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>> >>> Modification involves: >>> >>> - build set of entities in sync group >>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>> >>> >>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, >>> what they have done. >>> >>> Marc Morin >>> Emforium Group Inc. >>> ALL-IN Software™ >>> 519-772-6824 ext 201 >>> [hidden email] >>> >>> >>> >> >> > > |
|
|
That's great Jacques, I'm glad someone is coordinating and looking into things with it. -David On Oct 27, 2009, at 1:22 PM, Jacques Le Roux wrote: > Marc David, > > Sounds like David took the time to read you message... > > I remains, that we need to clarify what is working and what is not. > I should work on this these next days. > > Jacques > > From: "David E Jones" <[hidden email]> >> >> Marc, >> >> Actually this is a separate topic from what I've seen in the other >> thread. It would be good to read the other thread, but I'd >> recommend taking everything there with a grain of salt. There >> seems to be some misunderstanding in a lot of the assessments of >> what the Entity Sync does and what the issues are with it (ie >> things that make it slow, the constraints it has trying to >> reproduce time order of database changes over multiple tables, etc). >> >> In any case, yes, it is correct that the current Entity Sync stuff >> does not have filtering on a record level (other than by time), >> and also has nothing to follow a graph of related records to >> include desired information. >> >> -David >> >> >> On Oct 27, 2009, at 8:24 AM, Marc Morin wrote: >> >>> Rookie error... apologies >>> >>> ----- Original Message ----- >>> From: "Jacques Le Roux" <[hidden email]> >>> To: [hidden email] >>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada >>> Eastern >>> Subject: Re: Entity Sync >>> >>> Hi Marc, >>> >>> Did you follow recent threads on this subject ? >>> >>> Jacques >>> >>> From: "Marc Morin" <[hidden email]> >>>> We have a couple of use cases where we need to sync a subset of >>>> the information in one database and mirror it into another. >>>> >>>> Been looking at the entity sync capability that is present in >>>> ofbiz. Very good start, ability to build a sync set that is a >>>> collection of entities to sync (pull or push) between delegators >>>> (or service). >>>> >>>> In my case, I need to create the nest of entities to consider >>>> for the sync, but I need to restrict the rows from some starting >>>> point. >>>> >>>> Use case 1: Want to import a collection of parties from one >>>> system along with their relationships between the set of parties >>>> and >>>> their contact info, notes, preferences, etc.... >>>> >>>> Use case 2: Create a product category, rollups, and then >>>> associate products to that category. Want to publish this >>>> category, >>>> products, and subset of pricing, product content, features, >>>> etc.... to other instances. >>>> >>>> Looks like a modification to entity sync is required to do this? >>>> or am i mistaken. >>>> >>>> Modification involves: >>>> >>>> - build set of entities in sync group >>>> - add condition restrictions on possibly any number of entities >>>> for candidate rows. (category_id = 'x') >>>> - "soft" inclusion by foreign key reference from a row in the >>>> set to one that is not explicit. >>>> - alter row for foreign key references outside of entity set to >>>> null or to rows that cannot be replicated. >>>> >>>> >>>> I'm sure more precision is needed here and will to do the work. >>>> Just wanted to see if others have similar use cases and if so, >>>> what they have done. >>>> >>>> Marc Morin >>>> Emforium Group Inc. >>>> ALL-IN Software™ >>>> 519-772-6824 ext 201 >>>> [hidden email] >>>> >>>> >>>> >>> >>> >> > > |
|
Administrator
|
From: "David E Jones" <[hidden email]>
> That's great Jacques, I'm glad someone is coordinating and looking into things with it. I appreciate the compliment, I have no escape now ;o) Jacques > -David > > > On Oct 27, 2009, at 1:22 PM, Jacques Le Roux wrote: > >> Marc David, >> >> Sounds like David took the time to read you message... >> >> I remains, that we need to clarify what is working and what is not. I should work on this these next days. >> >> Jacques >> >> From: "David E Jones" <[hidden email]> >>> >>> Marc, >>> >>> Actually this is a separate topic from what I've seen in the other thread. It would be good to read the other thread, but I'd >>> recommend taking everything there with a grain of salt. There seems to be some misunderstanding in a lot of the assessments >>> of what the Entity Sync does and what the issues are with it (ie things that make it slow, the constraints it has trying to >>> reproduce time order of database changes over multiple tables, etc). >>> >>> In any case, yes, it is correct that the current Entity Sync stuff does not have filtering on a record level (other than by >>> time), and also has nothing to follow a graph of related records to include desired information. >>> >>> -David >>> >>> >>> On Oct 27, 2009, at 8:24 AM, Marc Morin wrote: >>> >>>> Rookie error... apologies >>>> >>>> ----- Original Message ----- >>>> From: "Jacques Le Roux" <[hidden email]> >>>> To: [hidden email] >>>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >>>> Subject: Re: Entity Sync >>>> >>>> Hi Marc, >>>> >>>> Did you follow recent threads on this subject ? >>>> >>>> Jacques >>>> >>>> From: "Marc Morin" <[hidden email]> >>>>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>>>> >>>>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>>>> collection of entities to sync (pull or push) between delegators (or service). >>>>> >>>>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some >>>>> starting >>>>> point. >>>>> >>>>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties >>>>> and >>>>> their contact info, notes, preferences, etc.... >>>>> >>>>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this >>>>> category, >>>>> products, and subset of pricing, product content, features, etc.... to other instances. >>>>> >>>>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>>>> >>>>> Modification involves: >>>>> >>>>> - build set of entities in sync group >>>>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>>>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>>>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>>>> >>>>> >>>>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if >>>>> so, >>>>> what they have done. >>>>> >>>>> Marc Morin >>>>> Emforium Group Inc. >>>>> ALL-IN Software™ >>>>> 519-772-6824 ext 201 >>>>> [hidden email] >>>>> >>>>> >>>>> >>>> >>>> >>> >> >> > > |
|
|
Yup, we all thank you :)
I'll send you my entity sync setup and some notes and stuff i've discovered by the end of the week. In general it can be re-worked a bit and still be a very useful tool so users can choose whether to use it or use some db level synchronization. -- Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: Entity Sync Date: Tue, 27 Oct 2009 21:05:51 +0100 From: "David E Jones" <[hidden email]> > That's great Jacques, I'm glad someone is coordinating and looking into things with it. I appreciate the compliment, I have no escape now ;o) Jacques > -David > > > On Oct 27, 2009, at 1:22 PM, Jacques Le Roux wrote: > >> Marc David, >> >> Sounds like David took the time to read you message... >> >> I remains, that we need to clarify what is working and what is not. I should work on this these next days. >> >> Jacques >> >> From: "David E Jones" <[hidden email]> >>> >>> Marc, >>> >>> Actually this is a separate topic from what I've seen in the other thread. It would be good to read the other thread, but I'd >>> recommend taking everything there with a grain of salt. There seems to be some misunderstanding in a lot of the assessments >>> of what the Entity Sync does and what the issues are with it (ie things that make it slow, the constraints it has trying to >>> reproduce time order of database changes over multiple tables, etc). >>> >>> In any case, yes, it is correct that the current Entity Sync stuff does not have filtering on a record level (other than by >>> time), and also has nothing to follow a graph of related records to include desired information. >>> >>> -David >>> >>> >>> On Oct 27, 2009, at 8:24 AM, Marc Morin wrote: >>> >>>> Rookie error... apologies >>>> >>>> ----- Original Message ----- >>>> From: "Jacques Le Roux" <[hidden email]> >>>> To: [hidden email] >>>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >>>> Subject: Re: Entity Sync >>>> >>>> Hi Marc, >>>> >>>> Did you follow recent threads on this subject ? >>>> >>>> Jacques >>>> >>>> From: "Marc Morin" <[hidden email]> >>>>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>>>> >>>>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>>>> collection of entities to sync (pull or push) between delegators (or service). >>>>> >>>>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some >>>>> starting >>>>> point. >>>>> >>>>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties >>>>> and >>>>> their contact info, notes, preferences, etc.... >>>>> >>>>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this >>>>> category, >>>>> products, and subset of pricing, product content, features, etc.... to other instances. >>>>> >>>>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>>>> >>>>> Modification involves: >>>>> >>>>> - build set of entities in sync group >>>>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>>>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>>>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>>>> >>>>> >>>>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if >>>>> so, >>>>> what they have done. >>>>> >>>>> Marc Morin >>>>> Emforium Group Inc. >>>>> ALL-IN Software™ >>>>> 519-772-6824 ext 201 >>>>> [hidden email] >>>>> >>>>> >>>>> >>>> >>>> >>> >> >> > > |
|
|
We could also discuss another idea:
Using the record's timestamp for generating a graph of sync entities is tricky. We need pretty precise system clock ntp sync and cpu time for ofbiz to generate the graph. Another approach might be to implement triggers in entity engine ( if that feature does not exist already ) and create a transaction log. I use the same approach but on a DB level - using DB triggers. The clients remember the last ID of the transaction log entry they have successfully reached so they just ask the server - give me everything after it. The transaction log approach would eliminate the system clock issues and graph generation. -- deyan -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: Jacques Le Roux <[hidden email]> Cc: [hidden email] Subject: Re: Entity Sync Date: Wed, 28 Oct 2009 09:31:01 +0200 Yup, we all thank you :) I'll send you my entity sync setup and some notes and stuff i've discovered by the end of the week. In general it can be re-worked a bit and still be a very useful tool so users can choose whether to use it or use some db level synchronization. -- Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: Entity Sync Date: Tue, 27 Oct 2009 21:05:51 +0100 From: "David E Jones" <[hidden email]> > That's great Jacques, I'm glad someone is coordinating and looking into things with it. I appreciate the compliment, I have no escape now ;o) Jacques > -David > > > On Oct 27, 2009, at 1:22 PM, Jacques Le Roux wrote: > >> Marc David, >> >> Sounds like David took the time to read you message... >> >> I remains, that we need to clarify what is working and what is not. I should work on this these next days. >> >> Jacques >> >> From: "David E Jones" <[hidden email]> >>> >>> Marc, >>> >>> Actually this is a separate topic from what I've seen in the other thread. It would be good to read the other thread, but I'd >>> recommend taking everything there with a grain of salt. There seems to be some misunderstanding in a lot of the assessments >>> of what the Entity Sync does and what the issues are with it (ie things that make it slow, the constraints it has trying to >>> reproduce time order of database changes over multiple tables, etc). >>> >>> In any case, yes, it is correct that the current Entity Sync stuff does not have filtering on a record level (other than by >>> time), and also has nothing to follow a graph of related records to include desired information. >>> >>> -David >>> >>> >>> On Oct 27, 2009, at 8:24 AM, Marc Morin wrote: >>> >>>> Rookie error... apologies >>>> >>>> ----- Original Message ----- >>>> From: "Jacques Le Roux" <[hidden email]> >>>> To: [hidden email] >>>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >>>> Subject: Re: Entity Sync >>>> >>>> Hi Marc, >>>> >>>> Did you follow recent threads on this subject ? >>>> >>>> Jacques >>>> >>>> From: "Marc Morin" <[hidden email]> >>>>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>>>> >>>>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>>>> collection of entities to sync (pull or push) between delegators (or service). >>>>> >>>>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some >>>>> starting >>>>> point. >>>>> >>>>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties >>>>> and >>>>> their contact info, notes, preferences, etc.... >>>>> >>>>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this >>>>> category, >>>>> products, and subset of pricing, product content, features, etc.... to other instances. >>>>> >>>>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>>>> >>>>> Modification involves: >>>>> >>>>> - build set of entities in sync group >>>>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>>>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>>>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>>>> >>>>> >>>>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if >>>>> so, >>>>> what they have done. >>>>> >>>>> Marc Morin >>>>> Emforium Group Inc. >>>>> ALL-IN Software™ >>>>> 519-772-6824 ext 201 >>>>> [hidden email] >>>>> >>>>> >>>>> >>>> >>>> >>> >> >> > > |
|
|
In reply to this post by Marc Morin
Yes, i go in spurts on reading this group. I just thought I'd drop on note on an issue that I am starting to investigate. I DIDN'T take the time (sorry) to read even a single email on the subject before posting. Imagine my embarrassment when I posted right in the middle of a heavily discussed topic on the exact topic I was posting on. Apologies.
Anyway, just wanted to put out a request to see if others see value in this "merged" rows type of sync. Essentially a set of content, a nest of rows (inter-related) are "published" from one instance and have multiple "slaves" for it. For the simpler case where the entire contents of tables are published and the slaves are "readonly", there are a number of db-level technologies that work very well for that,and I am not trying to replace/redo that case. So for this publish/subscribe model, doing this within Ofbiz will maintain the db independence, which is good. Ideally, there are three modes of updates: synchronous, asynchronous and batch. (Asynchronous and batch are very closely related and likely implemented the same way). We only need the asynch/batch mode. This could be done via: 1- eeca gathering up changes, published in master db as log, then send to slave for reply. 2- time base (like current sync) For our use case, either approach is fine, but since #2 is already in the entity-sync, we'd be leaning that way for now. Our application space is such that all instances of our databases are available to all app servers via a separate delegator, so, RMI/SOAP, and other issue about getting the change set from the master to slave application is much simplified. Since we'd be merging rows from multiple masters into a single table, we configure the sequenced-id-prefix for each instance to be different, generating unique ids. Marc Morin Emforium Group Inc. ALL-IN Software™ 519-772-6824 ext 201 [hidden email] ----- Original Message ----- From: "Jacques Le Roux" <[hidden email]> To: [hidden email] Sent: Tuesday, October 27, 2009 1:35:48 PM GMT -05:00 US/Canada Eastern Subject: Re: Entity Sync Do you mean it was a simple error on your side (sorry I did not take the time to read all yet ;o) ? Jacques From: "Marc Morin" <[hidden email]> > Rookie error... apologies > > ----- Original Message ----- > From: "Jacques Le Roux" <[hidden email]> > To: [hidden email] > Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern > Subject: Re: Entity Sync > > Hi Marc, > > Did you follow recent threads on this subject ? > > Jacques > > From: "Marc Morin" <[hidden email]> >> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >> >> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >> collection of entities to sync (pull or push) between delegators (or service). >> >> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting >> point. >> >> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and >> their contact info, notes, preferences, etc.... >> >> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >> products, and subset of pricing, product content, features, etc.... to other instances. >> >> Looks like a modification to entity sync is required to do this? or am i mistaken. >> >> Modification involves: >> >> - build set of entities in sync group >> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >> >> >> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, >> what they have done. >> >> Marc Morin >> Emforium Group Inc. >> ALL-IN Software™ >> 519-772-6824 ext 201 >> [hidden email] >> >> >> > > > |
|
Administrator
|
I have opened https://issues.apache.org/jira/browse/OFBIZ-3333 (what a number, though 333 is more symbolic :o)
More to come... Jacques From: "Marc Morin" <[hidden email]> > Yes, i go in spurts on reading this group. I just thought I'd drop on note on an issue that I am starting to investigate. I > DIDN'T take the time (sorry) to read even a single email on the subject before posting. Imagine my embarrassment when I posted > right in the middle of a heavily discussed topic on the exact topic I was posting on. Apologies. > > Anyway, just wanted to put out a request to see if others see value in this "merged" rows type of sync. Essentially a set of > content, a nest of rows (inter-related) are "published" from one instance and have multiple "slaves" for it. For the simpler > case where the entire contents of tables are published and the slaves are "readonly", there are a number of db-level technologies > that work very well for that,and I am not trying to replace/redo that case. > > So for this publish/subscribe model, doing this within Ofbiz will maintain the db independence, which is good. Ideally, there > are three modes of updates: synchronous, asynchronous and batch. (Asynchronous and batch are very closely related and likely > implemented the same way). We only need the asynch/batch mode. > > This could be done via: > > 1- eeca gathering up changes, published in master db as log, then send to slave for reply. > 2- time base (like current sync) > > For our use case, either approach is fine, but since #2 is already in the entity-sync, we'd be leaning that way for now. Our > application space is such that all instances of our databases are available to all app servers via a separate delegator, so, > RMI/SOAP, and other issue about getting the change set from the master to slave application is much simplified. > > Since we'd be merging rows from multiple masters into a single table, we configure the sequenced-id-prefix for each instance to be > different, generating unique ids. > > > Marc Morin > Emforium Group Inc. > ALL-IN Software™ > 519-772-6824 ext 201 > [hidden email] > > > > ----- Original Message ----- > From: "Jacques Le Roux" <[hidden email]> > To: [hidden email] > Sent: Tuesday, October 27, 2009 1:35:48 PM GMT -05:00 US/Canada Eastern > Subject: Re: Entity Sync > > Do you mean it was a simple error on your side (sorry I did not take the time to read all yet ;o) ? > > Jacques > > From: "Marc Morin" <[hidden email]> >> Rookie error... apologies >> >> ----- Original Message ----- >> From: "Jacques Le Roux" <[hidden email]> >> To: [hidden email] >> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >> Subject: Re: Entity Sync >> >> Hi Marc, >> >> Did you follow recent threads on this subject ? >> >> Jacques >> >> From: "Marc Morin" <[hidden email]> >>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>> >>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>> collection of entities to sync (pull or push) between delegators (or service). >>> >>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting >>> point. >>> >>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and >>> their contact info, notes, preferences, etc.... >>> >>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >>> products, and subset of pricing, product content, features, etc.... to other instances. >>> >>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>> >>> Modification involves: >>> >>> - build set of entities in sync group >>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>> >>> >>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, >>> what they have done. >>> >>> Marc Morin >>> Emforium Group Inc. >>> ALL-IN Software™ >>> 519-772-6824 ext 201 >>> [hidden email] >>> >>> >>> >> >> >> > > > |
|
Administrator
|
One question: in case of crash (be the net, the server or a POS - power outage for instance) should the
resetEntitySyncStatusToNotStarted service be run only from the server? Is it enough? Because most of the time, in my case, the problem comes from the net and unfortunately my client uses an IBM POS system which needs IBM jdk for the POS to run. But (without tweaks?) the IBM jdk is not able to run OFBiz in standard mode. So I can't use startofbizboth to have concurrently webtools and the POS GUI. Hope it's clear Jacques From: "Jacques Le Roux" <[hidden email]> >I have opened https://issues.apache.org/jira/browse/OFBIZ-3333 (what a number, though 333 is more symbolic :o) > More to come... > > Jacques > > From: "Marc Morin" <[hidden email]> >> Yes, i go in spurts on reading this group. I just thought I'd drop on note on an issue that I am starting to investigate. I >> DIDN'T take the time (sorry) to read even a single email on the subject before posting. Imagine my embarrassment when I posted >> right in the middle of a heavily discussed topic on the exact topic I was posting on. Apologies. >> >> Anyway, just wanted to put out a request to see if others see value in this "merged" rows type of sync. Essentially a set of >> content, a nest of rows (inter-related) are "published" from one instance and have multiple "slaves" for it. For the simpler >> case where the entire contents of tables are published and the slaves are "readonly", there are a number of db-level technologies >> that work very well for that,and I am not trying to replace/redo that case. >> >> So for this publish/subscribe model, doing this within Ofbiz will maintain the db independence, which is good. Ideally, there >> are three modes of updates: synchronous, asynchronous and batch. (Asynchronous and batch are very closely related and likely >> implemented the same way). We only need the asynch/batch mode. >> >> This could be done via: >> >> 1- eeca gathering up changes, published in master db as log, then send to slave for reply. >> 2- time base (like current sync) >> >> For our use case, either approach is fine, but since #2 is already in the entity-sync, we'd be leaning that way for now. Our >> application space is such that all instances of our databases are available to all app servers via a separate delegator, so, >> RMI/SOAP, and other issue about getting the change set from the master to slave application is much simplified. >> >> Since we'd be merging rows from multiple masters into a single table, we configure the sequenced-id-prefix for each instance to >> be different, generating unique ids. >> >> >> Marc Morin >> Emforium Group Inc. >> ALL-IN Software™ >> 519-772-6824 ext 201 >> [hidden email] >> >> >> >> ----- Original Message ----- >> From: "Jacques Le Roux" <[hidden email]> >> To: [hidden email] >> Sent: Tuesday, October 27, 2009 1:35:48 PM GMT -05:00 US/Canada Eastern >> Subject: Re: Entity Sync >> >> Do you mean it was a simple error on your side (sorry I did not take the time to read all yet ;o) ? >> >> Jacques >> >> From: "Marc Morin" <[hidden email]> >>> Rookie error... apologies >>> >>> ----- Original Message ----- >>> From: "Jacques Le Roux" <[hidden email]> >>> To: [hidden email] >>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >>> Subject: Re: Entity Sync >>> >>> Hi Marc, >>> >>> Did you follow recent threads on this subject ? >>> >>> Jacques >>> >>> From: "Marc Morin" <[hidden email]> >>>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>>> >>>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>>> collection of entities to sync (pull or push) between delegators (or service). >>>> >>>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting >>>> point. >>>> >>>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties >>>> and >>>> their contact info, notes, preferences, etc.... >>>> >>>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >>>> products, and subset of pricing, product content, features, etc.... to other instances. >>>> >>>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>>> >>>> Modification involves: >>>> >>>> - build set of entities in sync group >>>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>>> >>>> >>>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, >>>> what they have done. >>>> >>>> Marc Morin >>>> Emforium Group Inc. >>>> ALL-IN Software™ >>>> 519-772-6824 ext 201 >>>> [hidden email] >>>> >>>> >>>> >>> >>> >>> >> >> >> > > |
|
|
This is a tough one. My concern is that we're restricted by the error handling of the lower level tools, unless we make significant changes to how this is coded (and I'm not sure what those changes would have to look like without doing some research). -David On Dec 12, 2009, at 6:24 PM, Jacques Le Roux wrote: > One question: in case of crash (be the net, the server or a POS - power outage for instance) should the resetEntitySyncStatusToNotStarted service be run only from the server? Is it enough? > Because most of the time, in my case, the problem comes from the net and unfortunately my client uses an IBM POS system which needs IBM jdk for the POS to run. But (without tweaks?) the IBM jdk is not able to run OFBiz in standard mode. So I can't use startofbizboth to have concurrently webtools and the POS GUI. > > Hope it's clear > > Jacques > > From: "Jacques Le Roux" <[hidden email]> >> I have opened https://issues.apache.org/jira/browse/OFBIZ-3333 (what a number, though 333 is more symbolic :o) >> More to come... >> >> Jacques >> >> From: "Marc Morin" <[hidden email]> >>> Yes, i go in spurts on reading this group. I just thought I'd drop on note on an issue that I am starting to investigate. I DIDN'T take the time (sorry) to read even a single email on the subject before posting. Imagine my embarrassment when I posted right in the middle of a heavily discussed topic on the exact topic I was posting on. Apologies. >>> >>> Anyway, just wanted to put out a request to see if others see value in this "merged" rows type of sync. Essentially a set of content, a nest of rows (inter-related) are "published" from one instance and have multiple "slaves" for it. For the simpler case where the entire contents of tables are published and the slaves are "readonly", there are a number of db-level technologies that work very well for that,and I am not trying to replace/redo that case. >>> >>> So for this publish/subscribe model, doing this within Ofbiz will maintain the db independence, which is good. Ideally, there are three modes of updates: synchronous, asynchronous and batch. (Asynchronous and batch are very closely related and likely implemented the same way). We only need the asynch/batch mode. >>> >>> This could be done via: >>> >>> 1- eeca gathering up changes, published in master db as log, then send to slave for reply. >>> 2- time base (like current sync) >>> >>> For our use case, either approach is fine, but since #2 is already in the entity-sync, we'd be leaning that way for now. Our application space is such that all instances of our databases are available to all app servers via a separate delegator, so, RMI/SOAP, and other issue about getting the change set from the master to slave application is much simplified. >>> >>> Since we'd be merging rows from multiple masters into a single table, we configure the sequenced-id-prefix for each instance to be different, generating unique ids. >>> >>> >>> Marc Morin >>> Emforium Group Inc. >>> ALL-IN Software™ >>> 519-772-6824 ext 201 >>> [hidden email] >>> >>> >>> >>> ----- Original Message ----- >>> From: "Jacques Le Roux" <[hidden email]> >>> To: [hidden email] >>> Sent: Tuesday, October 27, 2009 1:35:48 PM GMT -05:00 US/Canada Eastern >>> Subject: Re: Entity Sync >>> >>> Do you mean it was a simple error on your side (sorry I did not take the time to read all yet ;o) ? >>> >>> Jacques >>> >>> From: "Marc Morin" <[hidden email]> >>>> Rookie error... apologies >>>> >>>> ----- Original Message ----- >>>> From: "Jacques Le Roux" <[hidden email]> >>>> To: [hidden email] >>>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >>>> Subject: Re: Entity Sync >>>> >>>> Hi Marc, >>>> >>>> Did you follow recent threads on this subject ? >>>> >>>> Jacques >>>> >>>> From: "Marc Morin" <[hidden email]> >>>>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>>>> >>>>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>>>> collection of entities to sync (pull or push) between delegators (or service). >>>>> >>>>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some starting >>>>> point. >>>>> >>>>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties and >>>>> their contact info, notes, preferences, etc.... >>>>> >>>>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >>>>> products, and subset of pricing, product content, features, etc.... to other instances. >>>>> >>>>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>>>> >>>>> Modification involves: >>>>> >>>>> - build set of entities in sync group >>>>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>>>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>>>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>>>> >>>>> >>>>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if so, >>>>> what they have done. >>>>> >>>>> Marc Morin >>>>> Emforium Group Inc. >>>>> ALL-IN Software™ >>>>> 519-772-6824 ext 201 >>>>> [hidden email] >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> > > |
|
Administrator
|
At least I was lucky this night. All seem to have run smoothly after I reset the all system. Is it right to think that the pull does
not occur before the push is ok? And then alternately I suppose? Or is done concurently and only based on time? I think the last assumption is correct finally. But as the 1st push was huge (for this system) it took at long time to proceed, maybe the reason of the my question above. No pull needs to be done for now anyway (but the job runs ok) I feel better :o) Jacques From: "David E Jones" <[hidden email]> > This is a tough one. My concern is that we're restricted by the error handling of the lower level tools, unless we make > significant changes to how this is coded (and I'm not sure what those changes would have to look like without doing some > research). > > -David > > > On Dec 12, 2009, at 6:24 PM, Jacques Le Roux wrote: > >> One question: in case of crash (be the net, the server or a POS - power outage for instance) should the >> resetEntitySyncStatusToNotStarted service be run only from the server? Is it enough? >> Because most of the time, in my case, the problem comes from the net and unfortunately my client uses an IBM POS system which >> needs IBM jdk for the POS to run. But (without tweaks?) the IBM jdk is not able to run OFBiz in standard mode. So I can't use >> startofbizboth to have concurrently webtools and the POS GUI. >> >> Hope it's clear >> >> Jacques >> >> From: "Jacques Le Roux" <[hidden email]> >>> I have opened https://issues.apache.org/jira/browse/OFBIZ-3333 (what a number, though 333 is more symbolic :o) >>> More to come... >>> >>> Jacques >>> >>> From: "Marc Morin" <[hidden email]> >>>> Yes, i go in spurts on reading this group. I just thought I'd drop on note on an issue that I am starting to investigate. I >>>> DIDN'T take the time (sorry) to read even a single email on the subject before posting. Imagine my embarrassment when I posted >>>> right in the middle of a heavily discussed topic on the exact topic I was posting on. Apologies. >>>> >>>> Anyway, just wanted to put out a request to see if others see value in this "merged" rows type of sync. Essentially a set of >>>> content, a nest of rows (inter-related) are "published" from one instance and have multiple "slaves" for it. For the simpler >>>> case where the entire contents of tables are published and the slaves are "readonly", there are a number of db-level >>>> technologies that work very well for that,and I am not trying to replace/redo that case. >>>> >>>> So for this publish/subscribe model, doing this within Ofbiz will maintain the db independence, which is good. Ideally, there >>>> are three modes of updates: synchronous, asynchronous and batch. (Asynchronous and batch are very closely related and likely >>>> implemented the same way). We only need the asynch/batch mode. >>>> >>>> This could be done via: >>>> >>>> 1- eeca gathering up changes, published in master db as log, then send to slave for reply. >>>> 2- time base (like current sync) >>>> >>>> For our use case, either approach is fine, but since #2 is already in the entity-sync, we'd be leaning that way for now. Our >>>> application space is such that all instances of our databases are available to all app servers via a separate delegator, so, >>>> RMI/SOAP, and other issue about getting the change set from the master to slave application is much simplified. >>>> >>>> Since we'd be merging rows from multiple masters into a single table, we configure the sequenced-id-prefix for each instance to >>>> be different, generating unique ids. >>>> >>>> >>>> Marc Morin >>>> Emforium Group Inc. >>>> ALL-IN Software™ >>>> 519-772-6824 ext 201 >>>> [hidden email] >>>> >>>> >>>> >>>> ----- Original Message ----- >>>> From: "Jacques Le Roux" <[hidden email]> >>>> To: [hidden email] >>>> Sent: Tuesday, October 27, 2009 1:35:48 PM GMT -05:00 US/Canada Eastern >>>> Subject: Re: Entity Sync >>>> >>>> Do you mean it was a simple error on your side (sorry I did not take the time to read all yet ;o) ? >>>> >>>> Jacques >>>> >>>> From: "Marc Morin" <[hidden email]> >>>>> Rookie error... apologies >>>>> >>>>> ----- Original Message ----- >>>>> From: "Jacques Le Roux" <[hidden email]> >>>>> To: [hidden email] >>>>> Sent: Tuesday, October 27, 2009 9:36:50 AM GMT -05:00 US/Canada Eastern >>>>> Subject: Re: Entity Sync >>>>> >>>>> Hi Marc, >>>>> >>>>> Did you follow recent threads on this subject ? >>>>> >>>>> Jacques >>>>> >>>>> From: "Marc Morin" <[hidden email]> >>>>>> We have a couple of use cases where we need to sync a subset of the information in one database and mirror it into another. >>>>>> >>>>>> Been looking at the entity sync capability that is present in ofbiz. Very good start, ability to build a sync set that is a >>>>>> collection of entities to sync (pull or push) between delegators (or service). >>>>>> >>>>>> In my case, I need to create the nest of entities to consider for the sync, but I need to restrict the rows from some >>>>>> starting >>>>>> point. >>>>>> >>>>>> Use case 1: Want to import a collection of parties from one system along with their relationships between the set of parties >>>>>> and >>>>>> their contact info, notes, preferences, etc.... >>>>>> >>>>>> Use case 2: Create a product category, rollups, and then associate products to that category. Want to publish this category, >>>>>> products, and subset of pricing, product content, features, etc.... to other instances. >>>>>> >>>>>> Looks like a modification to entity sync is required to do this? or am i mistaken. >>>>>> >>>>>> Modification involves: >>>>>> >>>>>> - build set of entities in sync group >>>>>> - add condition restrictions on possibly any number of entities for candidate rows. (category_id = 'x') >>>>>> - "soft" inclusion by foreign key reference from a row in the set to one that is not explicit. >>>>>> - alter row for foreign key references outside of entity set to null or to rows that cannot be replicated. >>>>>> >>>>>> >>>>>> I'm sure more precision is needed here and will to do the work. Just wanted to see if others have similar use cases and if >>>>>> so, >>>>>> what they have done. >>>>>> >>>>>> Marc Morin >>>>>> Emforium Group Inc. >>>>>> ALL-IN Software™ >>>>>> 519-772-6824 ext 201 >>>>>> [hidden email] >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> >>> >> >> > > |
«
Return to OFBiz - User
|
1 view|%1 views
| Free forum by Nabble | Edit this page |