database replication

[Bahaa Alamood]

Hello All,

We have a situation that we need to have more than 2 servers of ofbiz
running, server A, B, and C. The scenario is like this

1. Server A is the main server and it is online all the time with its
own database. some people do connect to that and make changes to the
data (create, delete, modify)

2. Server B is a server in one office (own database) and they do not
have a reliable internet connection so it goes offline some times while
the users in this office continue to use their local ofbiz

3. Server C is the same as server B with its own database as well

I have been looking at the bdr from 2nd Quadrant
https://2ndquadrant.com/en/resources/bdr/ to do the data
replicationamong the servers. I realize that this could create conflicts
in the primary keys of many things in the system. So I looked at
SequenceUtil.java and I can see if I change the stagger in getNextSeqId  
from 1 to 2 in one of the systems I can avoid this conflict in two of
them, but what about the third one? also in the above scenario what else
I need to look out for other than the sequences that might create conflicts?

--
Bahaaldin Al-amood
Tel: [phone]
International 001 540 632 1388
email: [hidden email]
Skype ID: bahaa.alamood

Arc Digital Solutions and Consultancy
www.arcdsc.com

This message contains information that may be confidential
and privileged to Arc digital Solutions and Consultancy, its partners, or customers.
Unauthorized use is strictly prohibited. Unless you are the
addressee (or authorized to receive mail for the addressee),
you should not use, copy or disclose to anyone this message
or any information contained in this message. If you have
received this message in error, please so advise the sender
by reply e-mail and delete this message. Thank you for your
cooperation.

[Jacques Le Roux]

The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)

You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example

Disclaimer: it's not very easy to understand and use...

HTH

Jacques


Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :

> Hello All,
>
> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>
> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data
> (create, delete, modify)
>
> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the
> users in this office continue to use their local ofbiz
>
> 3. Server C is the same as server B with its own database as well
>
> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize
> that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the
> stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above
> scenario what else I need to look out for other than the sequences that might create conflicts?
>

[Bahaa Alamood]

Hi Jaques,

thanks for the tip, I have seen this post before. I tried it today and
it seems to do the trick for the sequences. I think I do not need the
rest of the setup in the post because I want to use the brd which sync
the database.

I have a questions regarding the sync. So i have a scenario  as an
example where one user on one server changes a record and I have another
user on another server changed the same record. would the system be able
to handle this when they sync? will I see both changes as history?


On 1/17/2017 4:29 AM, Jacques Le Roux wrote:

> The now removed POS component used a solution for similar cases. This
> solution still exists and is reliable on a LAN ("less" on Internet)
>
> You can find the documentation at
> https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>
> Disclaimer: it's not very easy to understand and use...
>
> HTH
>
> Jacques
>
>
> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>> Hello All,
>>
>> We have a situation that we need to have more than 2 servers of ofbiz
>> running, server A, B, and C. The scenario is like this
>>
>> 1. Server A is the main server and it is online all the time with its
>> own database. some people do connect to that and make changes to the
>> data (create, delete, modify)
>>
>> 2. Server B is a server in one office (own database) and they do not
>> have a reliable internet connection so it goes offline some times
>> while the users in this office continue to use their local ofbiz
>>
>> 3. Server C is the same as server B with its own database as well
>>
>> I have been looking at the bdr from 2nd Quadrant
>> https://2ndquadrant.com/en/resources/bdr/ to do the data
>> replicationamong the servers. I realize that this could create
>> conflicts in the primary keys of many things in the system. So I
>> looked at SequenceUtil.java and I can see if I change the stagger in
>> getNextSeqId  from 1 to 2 in one of the systems I can avoid this
>> conflict in two of them, but what about the third one? also in the
>> above scenario what else I need to look out for other than the
>> sequences that might create conflicts?
>>
>

[marcopaul]

How expensive would it be to just upgrade the office’s internet connectivity? With my business hat on, that will almost always be MUCH cheaper than investing in a corner case of development that the rest of the community isn’t using anyway.

Take a look at the new generation of microwave data links available - you can now securely link sites at 1gbps+ point-to-point over 5-10 miles with 99.999% uptime with modest hardware investments (which against are always cheaper than trying to re-architect OFBiz).

—P

> On Jan 17, 2017, at 10:15 AM, Bahaa Alamood <[hidden email]> wrote:
>
> Hi Jaques,
>
> thanks for the tip, I have seen this post before. I tried it today and it seems to do the trick for the sequences. I think I do not need the rest of the setup in the post because I want to use the brd which sync the database.
>
> I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
>
>
> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>> The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)
>>
>> You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>
>> Disclaimer: it's not very easy to understand and use...
>>
>> HTH
>>
>> Jacques
>>
>>
>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>> Hello All,
>>>
>>> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>>>
>>> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data (create, delete, modify)
>>>
>>> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the users in this office continue to use their local ofbiz
>>>
>>> 3. Server C is the same as server B with its own database as well
>>>
>>> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above scenario what else I need to look out for other than the sequences that might create conflicts?
>>>
>>
>

[Bahaa Alamood]

Hi Paul,

Thanks for the advice :), but this is not the point here. The point is  
having such a solution will provide you  with the clustered environment
that will make your important business data always stored in more than
one machine (I think you  know how important that is) We can forget
about the remote office scenario and think cloud or vps type situation
where you  need this redundancy, you  will see that this scenario is
very  valuable. Another way to think about this when this remote office
is located in parts of the world where a Mbps cost over 300 USD a month
then this option is a valuable option, is it not?


On 1/17/2017 1:10 PM, Paul Mandeltort wrote:

> How expensive would it be to just upgrade the office’s internet connectivity? With my business hat on, that will almost always be MUCH cheaper than investing in a corner case of development that the rest of the community isn’t using anyway.
>
> Take a look at the new generation of microwave data links available - you can now securely link sites at 1gbps+ point-to-point over 5-10 miles with 99.999% uptime with modest hardware investments (which against are always cheaper than trying to re-architect OFBiz).
>
> —P
>
>> On Jan 17, 2017, at 10:15 AM, Bahaa Alamood <[hidden email]> wrote:
>>
>> Hi Jaques,
>>
>> thanks for the tip, I have seen this post before. I tried it today and it seems to do the trick for the sequences. I think I do not need the rest of the setup in the post because I want to use the brd which sync the database.
>>
>> I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
>>
>>
>> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>>> The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)
>>>
>>> You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>>
>>> Disclaimer: it's not very easy to understand and use...
>>>
>>> HTH
>>>
>>> Jacques
>>>
>>>
>>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>>> Hello All,
>>>>
>>>> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>>>>
>>>> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data (create, delete, modify)
>>>>
>>>> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the users in this office continue to use their local ofbiz
>>>>
>>>> 3. Server C is the same as server B with its own database as well
>>>>
>>>> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above scenario what else I need to look out for other than the sequences that might create conflicts?
>>>>

[marcopaul]

Honestly, probably not. There are several HA Postgres hosts nowadays that are very cost-effective. Amazon RDS is one, for example, and just a few clicks and boom you have a replicated fail-over Postgres instance available to you for practically nothing. Set up good backup scheme and you have a bulletproof database infrastructure.  

OFBiz has lots of areas that need attention (back-end user experience….), but scaling and cloud-hosting databases is a problem that’s being solved already by many other folks, so no sense in re-inventing the wheel here.

Bandwidth costs will continue to fall across the entire world. The cost of supporting custom/odd-ball features in OFBiz that the whole community doesn’t need will continue to rise as the software ages and increases technical debt.

—P
 

> On Jan 17, 2017, at 12:47 PM, Bahaa Alamood <[hidden email]> wrote:
>
> Hi Paul,
>
> Thanks for the advice :), but this is not the point here. The point is  having such a solution will provide you  with the clustered environment that will make your important business data always stored in more than one machine (I think you  know how important that is) We can forget about the remote office scenario and think cloud or vps type situation where you  need this redundancy, you  will see that this scenario is very  valuable. Another way to think about this when this remote office is located in parts of the world where a Mbps cost over 300 USD a month then this option is a valuable option, is it not?
>
>
> On 1/17/2017 1:10 PM, Paul Mandeltort wrote:
>> How expensive would it be to just upgrade the office’s internet connectivity? With my business hat on, that will almost always be MUCH cheaper than investing in a corner case of development that the rest of the community isn’t using anyway.
>>
>> Take a look at the new generation of microwave data links available - you can now securely link sites at 1gbps+ point-to-point over 5-10 miles with 99.999% uptime with modest hardware investments (which against are always cheaper than trying to re-architect OFBiz).
>>
>> —P
>>
>>> On Jan 17, 2017, at 10:15 AM, Bahaa Alamood <[hidden email]> wrote:
>>>
>>> Hi Jaques,
>>>
>>> thanks for the tip, I have seen this post before. I tried it today and it seems to do the trick for the sequences. I think I do not need the rest of the setup in the post because I want to use the brd which sync the database.
>>>
>>> I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
>>>
>>>
>>> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>>>> The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)
>>>>
>>>> You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>>>
>>>> Disclaimer: it's not very easy to understand and use...
>>>>
>>>> HTH
>>>>
>>>> Jacques
>>>>
>>>>
>>>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>>>> Hello All,
>>>>>
>>>>> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>>>>>
>>>>> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data (create, delete, modify)
>>>>>
>>>>> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the users in this office continue to use their local ofbiz
>>>>>
>>>>> 3. Server C is the same as server B with its own database as well
>>>>>
>>>>> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above scenario what else I need to look out for other than the sequences that might create conflicts?
>>>>>
>

[Mike Z]

BDR handles this exact scenario.  It will be an excellent choice for
ofbiz.  Most multi master solutions are trigger based. BDR can easily
handle the 800 tables that ofbiz has and it doesn't need a table to have a
primary key, nor does it use triggers. If two or more masters updates the
same table, the last update wins.  If a remote node is down, the BDR node
will save the WAL files until the remote node picks them up.

On Jan 17, 2017 8:16 AM, "Bahaa Alamood" <
[hidden email]> wrote:

> Hi Jaques,
>
> thanks for the tip, I have seen this post before. I tried it today and it
> seems to do the trick for the sequences. I think I do not need the rest of
> the setup in the post because I want to use the brd which sync the database.
>
> I have a questions regarding the sync. So i have a scenario  as an example
> where one user on one server changes a record and I have another user on
> another server changed the same record. would the system be able to handle
> this when they sync? will I see both changes as history?
>
>
> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>
>> The now removed POS component used a solution for similar cases. This
>> solution still exists and is reliable on a LAN ("less" on Internet)
>>
>> You can find the documentation at https://cwiki.apache.org/confl
>> uence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>
>> Disclaimer: it's not very easy to understand and use...
>>
>> HTH
>>
>> Jacques
>>
>>
>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>
>>> Hello All,
>>>
>>> We have a situation that we need to have more than 2 servers of ofbiz
>>> running, server A, B, and C. The scenario is like this
>>>
>>> 1. Server A is the main server and it is online all the time with its
>>> own database. some people do connect to that and make changes to the data
>>> (create, delete, modify)
>>>
>>> 2. Server B is a server in one office (own database) and they do not
>>> have a reliable internet connection so it goes offline some times while the
>>> users in this office continue to use their local ofbiz
>>>
>>> 3. Server C is the same as server B with its own database as well
>>>
>>> I have been looking at the bdr from 2nd Quadrant
>>> https://2ndquadrant.com/en/resources/bdr/ to do the data
>>> replicationamong the servers. I realize that this could create conflicts in
>>> the primary keys of many things in the system. So I looked at
>>> SequenceUtil.java and I can see if I change the stagger in getNextSeqId
>>> from 1 to 2 in one of the systems I can avoid this conflict in two of them,
>>> but what about the third one? also in the above scenario what else I need
>>> to look out for other than the sequences that might create conflicts?
>>>
>>>
>>
>

[Nicolas Malin-2]

Hi Bahaa, for your sequence pb like spot jacques and I thhink you
already ready the delegator prefix save your pb on all creation

|<delegator name=||"default"| |entity-model-reader=||"main"|
|entity-group-reader=||"main"| |entity-eca-reader=||"main"|
|||distributed-cache-clear-enabled=||"false"| |sequenced-id-prefix=||"A"||>|
|||<group-map group-name=||"org.ofbiz"|
|datasource-name=||"localpostgres"||/>|
|||<group-map group-name=||"org.ofbiz.olap"|
|datasource-name=||"localpostgres"||/>|
|</delegator>|

Le 17/01/2017 à 17:15, Bahaa Alamood a écrit :
> [...] I have a questions regarding the sync. So i have a scenario  as
> an example where one user on one server changes a record and I have
> another user on another server changed the same record. would the
> system be able to handle this when they sync? will I see both changes
> as history?
As most case the sync analyze the last updateDate and the newer win.

Nicolas

[Jacques Le Roux]

Le 21/01/2017 à 08:58, Nicolas Malin a écrit :

> Hi Bahaa, for your sequence pb like spot jacques and I thhink you already ready the delegator prefix save your pb on all creation
>
> |<delegator name=||"default"| |entity-model-reader=||"main"| |entity-group-reader=||"main"| |entity-eca-reader=||"main"|
> |||distributed-cache-clear-enabled=||"false"| |sequenced-id-prefix=||"A"||>|
> |||<group-map group-name=||"org.ofbiz"| |datasource-name=||"localpostgres"||/>|
> |||<group-map group-name=||"org.ofbiz.olap"| |datasource-name=||"localpostgres"||/>|
> |</delegator>|
>
> Le 17/01/2017 à 17:15, Bahaa Alamood a écrit :
>> [...] I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another
>> user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
> As most case the sync analyze the last updateDate and the newer win.
>
> Nicolas
>

Yes, thanks Nicolas

Since it's not done simultaneously and ms count collisions should not happen

Jacques

[Bahaa Alamood]

Dear All,

thanks for all the replies, yes, I tested it and it works. I am
deploying it now on 2 machines and my set up is like the:

server A is a VPS with OpenVPN server (this server has a Public IP)
which will be used for all people that are not local to the remote office

server B is a VM machine with  OpenVPN client that will be located in
the remote office and using the delegator sequence-id_prefix to avoid
the collisions in the primary keys.

The two machines use the bdr extension from 2nd quadrant with Postgresql
9.4 with bdr enabled in both machines.

I will keep testing and report back.


On 1/21/2017 5:04 AM, Jacques Le Roux wrote:

> Le 21/01/2017 à 08:58, Nicolas Malin a écrit :
>> Hi Bahaa, for your sequence pb like spot jacques and I thhink you
>> already ready the delegator prefix save your pb on all creation
>>
>> |<delegator name=||"default"| |entity-model-reader=||"main"|
>> |entity-group-reader=||"main"| |entity-eca-reader=||"main"|
>> |||distributed-cache-clear-enabled=||"false"|
>> |sequenced-id-prefix=||"A"||>|
>> |||<group-map group-name=||"org.ofbiz"|
>> |datasource-name=||"localpostgres"||/>|
>> |||<group-map group-name=||"org.ofbiz.olap"|
>> |datasource-name=||"localpostgres"||/>|
>> |</delegator>|
>>
>> Le 17/01/2017 à 17:15, Bahaa Alamood a écrit :
>>> [...] I have a questions regarding the sync. So i have a scenario  
>>> as an example where one user on one server changes a record and I
>>> have another user on another server changed the same record. would
>>> the system be able to handle this when they sync? will I see both
>>> changes as history?
>> As most case the sync analyze the last updateDate and the newer win.
>>
>> Nicolas
>>
> Yes, thanks Nicolas
>
> Since it's not done simultaneously and ms count collisions should not
> happen
>
> Jacques
>

[Jacques Le Roux]

Hi Paul,

I just wonder, are you using Amazon RDS in production? And if so how went the last big outage?

Thanks

Jacques


Le 17/01/2017 à 21:42, Paul Mandeltort a écrit :

> Honestly, probably not. There are several HA Postgres hosts nowadays that are very cost-effective. Amazon RDS is one, for example, and just a few clicks and boom you have a replicated fail-over Postgres instance available to you for practically nothing. Set up good backup scheme and you have a bulletproof database infrastructure.
>
> OFBiz has lots of areas that need attention (back-end user experience….), but scaling and cloud-hosting databases is a problem that’s being solved already by many other folks, so no sense in re-inventing the wheel here.
>
> Bandwidth costs will continue to fall across the entire world. The cost of supporting custom/odd-ball features in OFBiz that the whole community doesn’t need will continue to rise as the software ages and increases technical debt.
>
> —P
>  
>> On Jan 17, 2017, at 12:47 PM, Bahaa Alamood <[hidden email]> wrote:
>>
>> Hi Paul,
>>
>> Thanks for the advice :), but this is not the point here. The point is  having such a solution will provide you  with the clustered environment that will make your important business data always stored in more than one machine (I think you  know how important that is) We can forget about the remote office scenario and think cloud or vps type situation where you  need this redundancy, you  will see that this scenario is very  valuable. Another way to think about this when this remote office is located in parts of the world where a Mbps cost over 300 USD a month then this option is a valuable option, is it not?
>>
>>
>> On 1/17/2017 1:10 PM, Paul Mandeltort wrote:
>>> How expensive would it be to just upgrade the office’s internet connectivity? With my business hat on, that will almost always be MUCH cheaper than investing in a corner case of development that the rest of the community isn’t using anyway.
>>>
>>> Take a look at the new generation of microwave data links available - you can now securely link sites at 1gbps+ point-to-point over 5-10 miles with 99.999% uptime with modest hardware investments (which against are always cheaper than trying to re-architect OFBiz).
>>>
>>> —P
>>>
>>>> On Jan 17, 2017, at 10:15 AM, Bahaa Alamood <[hidden email]> wrote:
>>>>
>>>> Hi Jaques,
>>>>
>>>> thanks for the tip, I have seen this post before. I tried it today and it seems to do the trick for the sequences. I think I do not need the rest of the setup in the post because I want to use the brd which sync the database.
>>>>
>>>> I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
>>>>
>>>>
>>>> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>>>>> The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)
>>>>>
>>>>> You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>>>>
>>>>> Disclaimer: it's not very easy to understand and use...
>>>>>
>>>>> HTH
>>>>>
>>>>> Jacques
>>>>>
>>>>>
>>>>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>>>>> Hello All,
>>>>>>
>>>>>> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>>>>>>
>>>>>> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data (create, delete, modify)
>>>>>>
>>>>>> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the users in this office continue to use their local ofbiz
>>>>>>
>>>>>> 3. Server C is the same as server B with its own database as well
>>>>>>
>>>>>> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above scenario what else I need to look out for other than the sequences that might create conflicts?
>>>>>>
>
>

[marcopaul]

We actually haven’t migrated our live setup yet - we are finishing up a few other projects including getting some docker stuff sorted out to make our IT headaches a lot easier.

The outage only affected the S3 service which is separate from RDS, and as far as I know there was no data loss, just an outage where the services were offline for a bit.

We got a prototype setup on my hardware with Rancher container management going and it’s REALLY promising for a proper OFbiz Dev->Test->Live workflow with almost no headaches and distributed contractors/developers. I will shared our findings with the group once we get it all working.

RDS does provide automated backup services so even if the outage did affect it I don’t think there would have been any data loss.

—P

> On Mar 8, 2017, at 11:36 AM, Jacques Le Roux <[hidden email]> wrote:
>
> Hi Paul,
>
> I just wonder, are you using Amazon RDS in production? And if so how went the last big outage?
>
> Thanks
>
> Jacques
>
>
> Le 17/01/2017 à 21:42, Paul Mandeltort a écrit :
>> Honestly, probably not. There are several HA Postgres hosts nowadays that are very cost-effective. Amazon RDS is one, for example, and just a few clicks and boom you have a replicated fail-over Postgres instance available to you for practically nothing. Set up good backup scheme and you have a bulletproof database infrastructure.
>>
>> OFBiz has lots of areas that need attention (back-end user experience….), but scaling and cloud-hosting databases is a problem that’s being solved already by many other folks, so no sense in re-inventing the wheel here.
>>
>> Bandwidth costs will continue to fall across the entire world. The cost of supporting custom/odd-ball features in OFBiz that the whole community doesn’t need will continue to rise as the software ages and increases technical debt.
>>
>> —P
>>  
>>> On Jan 17, 2017, at 12:47 PM, Bahaa Alamood <[hidden email]> wrote:
>>>
>>> Hi Paul,
>>>
>>> Thanks for the advice :), but this is not the point here. The point is  having such a solution will provide you  with the clustered environment that will make your important business data always stored in more than one machine (I think you  know how important that is) We can forget about the remote office scenario and think cloud or vps type situation where you  need this redundancy, you  will see that this scenario is very  valuable. Another way to think about this when this remote office is located in parts of the world where a Mbps cost over 300 USD a month then this option is a valuable option, is it not?
>>>
>>>
>>> On 1/17/2017 1:10 PM, Paul Mandeltort wrote:
>>>> How expensive would it be to just upgrade the office’s internet connectivity? With my business hat on, that will almost always be MUCH cheaper than investing in a corner case of development that the rest of the community isn’t using anyway.
>>>>
>>>> Take a look at the new generation of microwave data links available - you can now securely link sites at 1gbps+ point-to-point over 5-10 miles with 99.999% uptime with modest hardware investments (which against are always cheaper than trying to re-architect OFBiz).
>>>>
>>>> —P
>>>>
>>>>> On Jan 17, 2017, at 10:15 AM, Bahaa Alamood <[hidden email]> wrote:
>>>>>
>>>>> Hi Jaques,
>>>>>
>>>>> thanks for the tip, I have seen this post before. I tried it today and it seems to do the trick for the sequences. I think I do not need the rest of the setup in the post because I want to use the brd which sync the database.
>>>>>
>>>>> I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
>>>>>
>>>>>
>>>>> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>>>>>> The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)
>>>>>>
>>>>>> You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>>>>>
>>>>>> Disclaimer: it's not very easy to understand and use...
>>>>>>
>>>>>> HTH
>>>>>>
>>>>>> Jacques
>>>>>>
>>>>>>
>>>>>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>>>>>> Hello All,
>>>>>>>
>>>>>>> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>>>>>>>
>>>>>>> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data (create, delete, modify)
>>>>>>>
>>>>>>> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the users in this office continue to use their local ofbiz
>>>>>>>
>>>>>>> 3. Server C is the same as server B with its own database as well
>>>>>>>
>>>>>>> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above scenario what else I need to look out for other than the sequences that might create conflicts?
>>>>>>>
>>
>>
>

[Jacques Le Roux]

Thanks Paul!

Jacques

Le 08/03/2017 à 17:49, Paul Mandeltort a écrit :

> We actually haven’t migrated our live setup yet - we are finishing up a few other projects including getting some docker stuff sorted out to make our IT headaches a lot easier.
>
> The outage only affected the S3 service which is separate from RDS, and as far as I know there was no data loss, just an outage where the services were offline for a bit.
>
> We got a prototype setup on my hardware with Rancher container management going and it’s REALLY promising for a proper OFbiz Dev->Test->Live workflow with almost no headaches and distributed contractors/developers. I will shared our findings with the group once we get it all working.
>
> RDS does provide automated backup services so even if the outage did affect it I don’t think there would have been any data loss.
>
> —P
>
>
>> On Mar 8, 2017, at 11:36 AM, Jacques Le Roux <[hidden email]> wrote:
>>
>> Hi Paul,
>>
>> I just wonder, are you using Amazon RDS in production? And if so how went the last big outage?
>>
>> Thanks
>>
>> Jacques
>>
>>
>> Le 17/01/2017 à 21:42, Paul Mandeltort a écrit :
>>> Honestly, probably not. There are several HA Postgres hosts nowadays that are very cost-effective. Amazon RDS is one, for example, and just a few clicks and boom you have a replicated fail-over Postgres instance available to you for practically nothing. Set up good backup scheme and you have a bulletproof database infrastructure.
>>>
>>> OFBiz has lots of areas that need attention (back-end user experience….), but scaling and cloud-hosting databases is a problem that’s being solved already by many other folks, so no sense in re-inventing the wheel here.
>>>
>>> Bandwidth costs will continue to fall across the entire world. The cost of supporting custom/odd-ball features in OFBiz that the whole community doesn’t need will continue to rise as the software ages and increases technical debt.
>>>
>>> —P
>>>  
>>>> On Jan 17, 2017, at 12:47 PM, Bahaa Alamood <[hidden email]> wrote:
>>>>
>>>> Hi Paul,
>>>>
>>>> Thanks for the advice :), but this is not the point here. The point is  having such a solution will provide you  with the clustered environment that will make your important business data always stored in more than one machine (I think you  know how important that is) We can forget about the remote office scenario and think cloud or vps type situation where you  need this redundancy, you  will see that this scenario is very  valuable. Another way to think about this when this remote office is located in parts of the world where a Mbps cost over 300 USD a month then this option is a valuable option, is it not?
>>>>
>>>>
>>>> On 1/17/2017 1:10 PM, Paul Mandeltort wrote:
>>>>> How expensive would it be to just upgrade the office’s internet connectivity? With my business hat on, that will almost always be MUCH cheaper than investing in a corner case of development that the rest of the community isn’t using anyway.
>>>>>
>>>>> Take a look at the new generation of microwave data links available - you can now securely link sites at 1gbps+ point-to-point over 5-10 miles with 99.999% uptime with modest hardware investments (which against are always cheaper than trying to re-architect OFBiz).
>>>>>
>>>>> —P
>>>>>
>>>>>> On Jan 17, 2017, at 10:15 AM, Bahaa Alamood <[hidden email]> wrote:
>>>>>>
>>>>>> Hi Jaques,
>>>>>>
>>>>>> thanks for the tip, I have seen this post before. I tried it today and it seems to do the trick for the sequences. I think I do not need the rest of the setup in the post because I want to use the brd which sync the database.
>>>>>>
>>>>>> I have a questions regarding the sync. So i have a scenario  as an example where one user on one server changes a record and I have another user on another server changed the same record. would the system be able to handle this when they sync? will I see both changes as history?
>>>>>>
>>>>>>
>>>>>> On 1/17/2017 4:29 AM, Jacques Le Roux wrote:
>>>>>>> The now removed POS component used a solution for similar cases. This solution still exists and is reliable on a LAN ("less" on Internet)
>>>>>>>
>>>>>>> You can find the documentation at https://cwiki.apache.org/confluence/display/OFBIZ/Sync+Setup+Notes+and+Example
>>>>>>>
>>>>>>> Disclaimer: it's not very easy to understand and use...
>>>>>>>
>>>>>>> HTH
>>>>>>>
>>>>>>> Jacques
>>>>>>>
>>>>>>>
>>>>>>> Le 17/01/2017 à 00:01, Bahaa Alamood a écrit :
>>>>>>>> Hello All,
>>>>>>>>
>>>>>>>> We have a situation that we need to have more than 2 servers of ofbiz running, server A, B, and C. The scenario is like this
>>>>>>>>
>>>>>>>> 1. Server A is the main server and it is online all the time with its own database. some people do connect to that and make changes to the data (create, delete, modify)
>>>>>>>>
>>>>>>>> 2. Server B is a server in one office (own database) and they do not have a reliable internet connection so it goes offline some times while the users in this office continue to use their local ofbiz
>>>>>>>>
>>>>>>>> 3. Server C is the same as server B with its own database as well
>>>>>>>>
>>>>>>>> I have been looking at the bdr from 2nd Quadrant https://2ndquadrant.com/en/resources/bdr/ to do the data replicationamong the servers. I realize that this could create conflicts in the primary keys of many things in the system. So I looked at SequenceUtil.java and I can see if I change the stagger in getNextSeqId  from 1 to 2 in one of the systems I can avoid this conflict in two of them, but what about the third one? also in the above scenario what else I need to look out for other than the sequences that might create conflicts?
>>>>>>>>
>>>
>
>