EntitySync RMI error
Locked
 
|
Hi guys,
I'm trying to configure RMI entity sync. I'm following http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example What I've done so far: 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 ( as per the example ). 3) I've imported the entity sync groups, SandJobs, etc. However when the sync starts ( on the POS instance ) I get the following error: Exception calling remote pull and report EntitySync service with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) Exception: org.ofbiz.service.GenericServiceException Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) ---- cause --------------------------------------------------------------------- Exception: java.rmi.ConnectException Message: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect ---- cause --------------------------------------------------------------------- Exception: java.net.ConnectException Message: Connection refused: connect ---- stack trace --------------------------------------------------------------- It insists connecting to 127.0.0.1 no matter what I type in serviceengine.xml. Any help would be appreciated :) Thanks in advance, Deyan |
|
|
During startup I get:
2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ AbstractEngine.java:73 :INFO ] Loaded Service Locations : [main-rmi=rmi://127.0.0. 1:1099/RMIDispatcher, main-http=http://127.0.0.1:8080/webtools/control/httpService, entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. 0.0.1:8080/webtools/control/httpService] entity-sync-rmi seems to be ok ... -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: [hidden email] Subject: EntitySync RMI error Date: Sat, 26 Sep 2009 09:22:23 +0300 Hi guys, I'm trying to configure RMI entity sync. I'm following http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example What I've done so far: 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 ( as per the example ). 3) I've imported the entity sync groups, SandJobs, etc. However when the sync starts ( on the POS instance ) I get the following error: Exception calling remote pull and report EntitySync service with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) Exception: org.ofbiz.service.GenericServiceException Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) ---- cause --------------------------------------------------------------------- Exception: java.rmi.ConnectException Message: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect ---- cause --------------------------------------------------------------------- Exception: java.net.ConnectException Message: Connection refused: connect ---- stack trace --------------------------------------------------------------- It insists connecting to 127.0.0.1 no matter what I type in serviceengine.xml. Any help would be appreciated :) Thanks in advance, Deyan |
|
|
Got it !
Seems that the previous exception was coming from the MCS ( server ) . When I stopped the server I started getting the exception I kind of like more :) Message: Exception calling remote pull and report EntitySync service with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene ricServiceException: RMI Error (Connection refused to host: 192.168.1.100; nested exception is: java.net.ConnectException: Connection refused: connect) So the problem was at the server side - RMIIF env var in startofbiz.sh wasn't set , so ofbiz was trying to get RMI host IP by resolving the hostname . The misleading things for me was that: 1) no any sign in the MCS logs that somebody is trying to connect and the MCS itself can not connect to its own RMI registry. 2) no any difference in the exceptions text on the POS ( client ) side: the 2 exceptions ( local one - can not connect to 192.168.1.100 and the remote one - can not connect to 127.0.0.1 ) look the same :) So issue is solved, sorry for bothering :) -- deyan -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: [hidden email] Subject: Re: EntitySync RMI error Date: Sat, 26 Sep 2009 09:52:55 +0300 During startup I get: 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ AbstractEngine.java:73 :INFO ] Loaded Service Locations : [main-rmi=rmi://127.0.0. 1:1099/RMIDispatcher, main-http=http://127.0.0.1:8080/webtools/control/httpService, entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. 0.0.1:8080/webtools/control/httpService] entity-sync-rmi seems to be ok ... -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: [hidden email] Subject: EntitySync RMI error Date: Sat, 26 Sep 2009 09:22:23 +0300 Hi guys, I'm trying to configure RMI entity sync. I'm following http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example What I've done so far: 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 ( as per the example ). 3) I've imported the entity sync groups, SandJobs, etc. However when the sync starts ( on the POS instance ) I get the following error: Exception calling remote pull and report EntitySync service with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) Exception: org.ofbiz.service.GenericServiceException Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) ---- cause --------------------------------------------------------------------- Exception: java.rmi.ConnectException Message: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect ---- cause --------------------------------------------------------------------- Exception: java.net.ConnectException Message: Connection refused: connect ---- stack trace --------------------------------------------------------------- It insists connecting to 127.0.0.1 no matter what I type in serviceengine.xml. Any help would be appreciated :) Thanks in advance, Deyan |
|
|
I found an issue though:
If the connection gets dropped during sync : - the client ( POS ) prints connection reset by peer - the server ( MCS ) sync status remains to running. - the next sync does not start because MCS complains there is another sync running already. "Reset run status" from webtools -> entity sync status helps. I'll investigate further and log a bug, although the solution seems pretty simple - looks like a swallowed IOException, which should be handled by re-setting the sync status. CHeers, DEyan -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: [hidden email] Subject: Re: EntitySync RMI error Date: Sat, 26 Sep 2009 11:22:55 +0300 Got it ! Seems that the previous exception was coming from the MCS ( server ) . When I stopped the server I started getting the exception I kind of like more :) Message: Exception calling remote pull and report EntitySync service with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene ricServiceException: RMI Error (Connection refused to host: 192.168.1.100; nested exception is: java.net.ConnectException: Connection refused: connect) So the problem was at the server side - RMIIF env var in startofbiz.sh wasn't set , so ofbiz was trying to get RMI host IP by resolving the hostname . The misleading things for me was that: 1) no any sign in the MCS logs that somebody is trying to connect and the MCS itself can not connect to its own RMI registry. 2) no any difference in the exceptions text on the POS ( client ) side: the 2 exceptions ( local one - can not connect to 192.168.1.100 and the remote one - can not connect to 127.0.0.1 ) look the same :) So issue is solved, sorry for bothering :) -- deyan -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: [hidden email] Subject: Re: EntitySync RMI error Date: Sat, 26 Sep 2009 09:52:55 +0300 During startup I get: 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ AbstractEngine.java:73 :INFO ] Loaded Service Locations : [main-rmi=rmi://127.0.0. 1:1099/RMIDispatcher, main-http=http://127.0.0.1:8080/webtools/control/httpService, entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. 0.0.1:8080/webtools/control/httpService] entity-sync-rmi seems to be ok ... -----Original Message----- From: Deyan Tsvetanov <[hidden email]> Reply-to: [hidden email] To: [hidden email] Subject: EntitySync RMI error Date: Sat, 26 Sep 2009 09:22:23 +0300 Hi guys, I'm trying to configure RMI entity sync. I'm following http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example What I've done so far: 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 ( as per the example ). 3) I've imported the entity sync groups, SandJobs, etc. However when the sync starts ( on the POS instance ) I get the following error: Exception calling remote pull and report EntitySync service with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) Exception: org.ofbiz.service.GenericServiceException Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect) ---- cause --------------------------------------------------------------------- Exception: java.rmi.ConnectException Message: Connection refused to host: 127.0.0.1; nested exception is: java.net.ConnectException: Connection refused: connect ---- cause --------------------------------------------------------------------- Exception: java.net.ConnectException Message: Connection refused: connect ---- stack trace --------------------------------------------------------------- It insists connecting to 127.0.0.1 no matter what I type in serviceengine.xml. Any help would be appreciated :) Thanks in advance, Deyan |
Administrator
|
Hi Deyan,
It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
|
|
Hi guys,
I've a question: <EntityGroupEntry entityGroupId="5500" entityOrPackage="org.ofbiz.product" applEnumId="ESIA_INCLUDE"/> Are the sub-packages going to be included in the sync ( e.g. org.ofbiz.product.facility , org.ofbiz.product.price, etc ) or they have to be referenced implicitly ? Looks like they aren't to me ... Thanks, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
EntitySync issues & suggestions ( was EntitySync RMI error )
|
|
In reply to this post by Jacques Le Roux
Hi Jacques and list,
After playing for almost 2 weeks with the entity sync feature I've come up with the conclusion that it still needs some work to get ready for production environment. Currently it is unreliable, and useless :( - There is a bug in the design. The MCS remembers the sync status and currently reached sync timestamp for each client. This is not a good approach imho. Why ? The client asks the server: have you got something for me ? Next: The MCS checks the last successful sync timestamp for that particular client and selects the entities that should be passed to the client. After successfully transferring the entities to the client over RMI the MCS assumes that the sync is successful and updates the last successful sync timestamp. The problem is that the client does not confirm the success of the operation. If for some reason the client is unable to update the entities ( either shutdown, power failure, DB error or whatever ) the MCS isn't aware of that and the next sync will not include the missed entities. There is no way to fix the skipped entities but re-creating the client's database from zero. That is the main issue actually. It could be easily solved however. The most easy ways are: 1) The client will remember the sync timestamp and will include it as an argument when asking the server for pull sync. This way the client will update the last successful sync timestamp after storing each entity. The server does not have to remember anything regarding the client's sync status. 2) After each sync session the server will wait for confirmation from the client before updating the last successful sync timestamp. If not updated than the sync process is considered to be unsuccessful and the timestamp will not be updated. I personally prefer approach 1) as in case of network error, power failure or whatever error the sync process will continue from the place it has stopped. - There are some swallowed exceptions. If a network problem occurs during the sync process the server's status remains "sync running" and the client is not able to sync anymore until the status is reset by an operator. - The last issue is the performance. Using RMI is an easy approach from developer's perspective but it is very very slow. I had a database with 5000 products. The initial sync took about 2 hours. The hardware is pretty good, the network connection between the MCS and the client ofbiz is about 50 mbits. What if we have 100 000 sales per day ? The sync process will take probably the entire night :) Here I would suggest transferring GenericEntity instances over RMI to be removed. Instead a regular XML could be used. Either over RMI or SOAP - does not matter. So in general the picture would be: 1) Server does not remember any timestamp for any client. 2) Client calls the pull entity sync RMI ( or soap ) method and passes the entity sync ID and the timestamp of the last successful sync as arguments. Including the timezone of course :) 3) The server generates an XML file for all the entities in that entity sync group using the timestamp provided by the client. 4) Client starts updating the entities described in the received XML line by line. After each entity is updated successfully the timestamp is also updated ( on the client side - the server does not remember any client timestamps ). 5) If a failure occurs in the middle of the update process what we'll have is a sync process completed to like 50%. With the correct timestamp. When the next sync job starts the client will use the timestamp and the server will generate a new XML from the place the sync has failed. Of course an optimization could be used - the XML will be stored in a new db table so the client can continue storing the entities without asking the server to generate something he has already done. Some improvements that could be also useful: 1) The client ofbiz instances have sequence-id-prefix. So when the server is accepting push syncs it could also check the prefixes of the PKs and disallow entities without the correct prefix in the primary key. That's just a cheap insurance against errors :) We don't want store A to make sales on behalf of store B just because somebody made an error while configuring the pos ofbiz instance. 2) A time synchronization mechanism could be also implemented. In general when syncing there is a requirement that the server's and client's clocks are in sync ( ntp ). We don't want to set the system clock in ofbiz, but we could implement a check if the clocks are in sync and refuse to start the synchronization in certain cases. 3) A new column could be added to all the entities: isFromEntitySync : flag. This way we could avoid cases in which the client is trying to push records to the server which were actually pulled by the server. Yes, I know the entity's created timestamp should help us avoid such cases, but when the sync process is interrupted things like that could happen. Otherwise for our current project we've implemented a DB level synchronization - using triggers for INSERT, UPDATE and DELETE and a stored procedure which syncs the clients with a MCS. It works much ( tens of times ) faster than ofbiz entity sync . But it is DB dependent. So after getting comments and suggestions to this e-mail I would like to invest some time in fixing all those issues with the ofbiz entity sync and switch to it when it's ready :) Thanks for your time, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
|
Hi Deyan n others
I have also worked & solved many sync problems in ofbiz in my application: 1) As you said while sync is in progress if server crashes, the status in EntitySync table gets in running state and we can't sync further until we manually change it even if we restart server it remains in running state To solve this problem: I created a new entity SyncStatus on client side to maintain the status of sync, in that ist status is Not Started, when service runEntitySync or runPullEntitySync starts I set the status of field as running and when service returns (whether fail or successful) I updated status to finish. I also added code, that if server restarts and if this field status is running change it to not started, By this I solved the above problem: *) If client crashes, then after restart, status in SyncStatus table will be Not Started. *) If server crashes, then service will be failed and status will be finished and no further problem. 2) There were other problems like if sync(push or pull) is running, and one other job of same service is also created then we get exception "An instance is already running" By using above status in SyncStatus table when A job of sync is running, I just cancelled out all other jobs of same type. 3) I also solved "Connection refused" problem by same method even I displayed message on screen rather than giving exception on console. 4) I did not get your point of updating time stamp, I did not get this problem because In push or pull sync the successful timestamp is updated on EntitySync table on client or MCS side only if sync is successful, if server crashes or sync fails this field is not updated, but yes there are some issues: *) If we have 2 hours of sync, n sync fails on last minute. then in next sync as sync successful status has not been revisesd, next sync will pick the status of previous one, as a solution I think we need to keep updating syn status in EntitySync table after particular intervals. but how to decide these intervals b'coz tables are linked with foreign key constriants. *) One problem with connection refused in push and pull case is that, In pull sync first connection is established then data is prepared for sync while in push sync first data is prepared and then conncetion is established. So in pull sync we get connection refused problem in earlier stage than push sync. We can say in push sync if there is no MCS (Or MCS IP address is wrong), Our code is doing efforts in preparing data that effort is of no use. 5) If we place order on POS side and try to perform push sync, some time I was getting random exceptions due to order header or Order Status table, I solved the problem. Explain: In placing order (or others) if you have used a database trigger (eecas) like if an entry is created on A table create an entry on B table. Now wile sync if MCS is also having same trigger then as a entry is created (by sync) on A table , trigger is fired and an entry for B is created. Now for one A, B will be having two entries that creates random exception So I want to ask, that mean for sync there should not be ant trigger on opposite side(means for push MCS and for pull POS), how we will maintain that ??? I did not get your concept of using XML file inspite of RMI. Could you please give some details regarding that???? Even I agree with the fact that RMI takes long time. Regards: Pankaj Jain ______________________________________________________________________ |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
Administrator
|
In reply to this post by Deyan Tsvetanov
Hi Deyan,
Did you work on this ? I will need to use EntitySync soon and I know you are not the 1st to complain about it (Si Chen did for instance). On the other hand, some seems to have used it successfully... Thanks Jacques ----- Original Message ----- From: Deyan Tsvetanov To: Jacques Le Roux ; [hidden email] Sent: Sunday, October 04, 2009 10:57 AM Subject: EntitySync issues & suggestions ( was EntitySync RMI error ) Hi Jacques and list, After playing for almost 2 weeks with the entity sync feature I've come up with the conclusion that it still needs some work to get ready for production environment. Currently it is unreliable, and useless :( - There is a bug in the design. The MCS remembers the sync status and currently reached sync timestamp for each client. This is not a good approach imho. Why ? The client asks the server: have you got something for me ? Next: The MCS checks the last successful sync timestamp for that particular client and selects the entities that should be passed to the client. After successfully transferring the entities to the client over RMI the MCS assumes that the sync is successful and updates the last successful sync timestamp. The problem is that the client does not confirm the success of the operation. If for some reason the client is unable to update the entities ( either shutdown, power failure, DB error or whatever ) the MCS isn't aware of that and the next sync will not include the missed entities. There is no way to fix the skipped entities but re-creating the client's database from zero. That is the main issue actually. It could be easily solved however. The most easy ways are: 1) The client will remember the sync timestamp and will include it as an argument when asking the server for pull sync. This way the client will update the last successful sync timestamp after storing each entity. The server does not have to remember anything regarding the client's sync status. 2) After each sync session the server will wait for confirmation from the client before updating the last successful sync timestamp. If not updated than the sync process is considered to be unsuccessful and the timestamp will not be updated. I personally prefer approach 1) as in case of network error, power failure or whatever error the sync process will continue from the place it has stopped. - There are some swallowed exceptions. If a network problem occurs during the sync process the server's status remains "sync running" and the client is not able to sync anymore until the status is reset by an operator. - The last issue is the performance. Using RMI is an easy approach from developer's perspective but it is very very slow. I had a database with 5000 products. The initial sync took about 2 hours. The hardware is pretty good, the network connection between the MCS and the client ofbiz is about 50 mbits. What if we have 100 000 sales per day ? The sync process will take probably the entire night :) Here I would suggest transferring GenericEntity instances over RMI to be removed. Instead a regular XML could be used. Either over RMI or SOAP - does not matter. So in general the picture would be: 1) Server does not remember any timestamp for any client. 2) Client calls the pull entity sync RMI ( or soap ) method and passes the entity sync ID and the timestamp of the last successful sync as arguments. Including the timezone of course :) 3) The server generates an XML file for all the entities in that entity sync group using the timestamp provided by the client. 4) Client starts updating the entities described in the received XML line by line. After each entity is updated successfully the timestamp is also updated ( on the client side - the server does not remember any client timestamps ). 5) If a failure occurs in the middle of the update process what we'll have is a sync process completed to like 50%. With the correct timestamp. When the next sync job starts the client will use the timestamp and the server will generate a new XML from the place the sync has failed. Of course an optimization could be used - the XML will be stored in a new db table so the client can continue storing the entities without asking the server to generate something he has already done. Some improvements that could be also useful: 1) The client ofbiz instances have sequence-id-prefix. So when the server is accepting push syncs it could also check the prefixes of the PKs and disallow entities without the correct prefix in the primary key. That's just a cheap insurance against errors :) We don't want store A to make sales on behalf of store B just because somebody made an error while configuring the pos ofbiz instance. 2) A time synchronization mechanism could be also implemented. In general when syncing there is a requirement that the server's and client's clocks are in sync ( ntp ). We don't want to set the system clock in ofbiz, but we could implement a check if the clocks are in sync and refuse to start the synchronization in certain cases. 3) A new column could be added to all the entities: isFromEntitySync : flag. This way we could avoid cases in which the client is trying to push records to the server which were actually pulled by the server. Yes, I know the entity's created timestamp should help us avoid such cases, but when the sync process is interrupted things like that could happen. Otherwise for our current project we've implemented a DB level synchronization - using triggers for INSERT, UPDATE and DELETE and a stored procedure which syncs the clients with a MCS. It works much ( tens of times ) faster than ofbiz entity sync . But it is DB dependent. So after getting comments and suggestions to this e-mail I would like to invest some time in fixing all those issues with the ofbiz entity sync and switch to it when it's ready :) Thanks for your time, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
Administrator
|
In reply to this post by Pankaj.Jain
Thanks Pankaj,
I had not the time to read all your message, very useful I guess. I will post all this (with Deyan's remark too) on Wiki after having done my own opinion... Jacques From: <[hidden email]> > Hi Deyan n others > > I have also worked & solved many sync problems in ofbiz in my application: > > 1) As you said while sync is in progress if server crashes, the status in > EntitySync table gets in running state and we can't sync further until we > manually change it even if we restart server it remains in running state > To solve this problem: I created a new entity SyncStatus on client side > to maintain the status of sync, in that ist status is Not Started, when > service runEntitySync or runPullEntitySync starts I set the status of > field as running and when service returns (whether fail or successful) I > updated status to finish. > I also added code, that if server restarts and if this field status is > running change it to not started, By this I solved the above problem: > > *) If client crashes, then after restart, status in SyncStatus table will > be Not Started. > *) If server crashes, then service will be failed and status will be > finished and no further problem. > > 2) There were other problems like if sync(push or pull) is running, and > one other job of same service is also created then we get exception "An > instance is already running" By using above status in SyncStatus table > when A job of sync is running, I just cancelled out all other jobs of same > type. > > 3) I also solved "Connection refused" problem by same method even I > displayed message on screen rather than giving exception on console. > > 4) I did not get your point of updating time stamp, I did not get this > problem because In push or pull sync the successful timestamp is updated > on EntitySync table on client or MCS side only if sync is successful, if > server crashes or sync fails this field is not updated, but yes there are > some issues: > *) If we have 2 hours of sync, n sync fails on last minute. then in next > sync as sync successful status has not been revisesd, next sync will pick > the status of previous one, as a solution I think we need to keep updating > syn status in EntitySync table after particular intervals. but how to > decide these intervals b'coz tables are linked with foreign key > constriants. > *) One problem with connection refused in push and pull case is that, In > pull sync first connection is established then data is prepared for sync > while in push sync first data is prepared and then conncetion is > established. So in pull sync we get connection refused problem in earlier > stage than push sync. We can say in push sync if there is no MCS (Or MCS > IP address is wrong), Our code is doing efforts in preparing data that > effort is of no use. > > 5) If we place order on POS side and try to perform push sync, some time I > was getting random exceptions due to order header or Order Status table, I > solved the problem. > Explain: In placing order (or others) if you have used a database trigger > (eecas) like if an entry is created on A table create an entry on B table. > Now wile sync if MCS is also having same trigger then as a entry is > created (by sync) on A table , trigger is fired and an entry for B is > created. Now for one A, B will be having two entries that creates random > exception > So I want to ask, that mean for sync there should not be ant trigger on > opposite side(means for push MCS and for pull POS), how we will maintain > that ??? > > I did not get your concept of using XML file inspite of RMI. Could you > please give some details regarding that???? Even I agree with the fact > that RMI takes long time. > > > Regards: > > Pankaj Jain > > > > ______________________________________________________________________ |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
|
In reply to this post by Jacques Le Roux
Hi Jacques,
no, I didn't work on fixing the entity sync issues. Instead I did my own sync tool using db triggers and stored procedures. The entity sync could be fixed, but it would require like two - three weeks of work. I am sorry I didn't post any details, i was kind of busy because of a missed deadline :) The entity sync is kind of useless in this state imho. Reasons: 1) Does not do any fail over - if the sync process is interrupted for any reason - like network outage, or ofbiz restart, or system freeze it will be considered as a success sync and won't continue from the point it has failed. Could be fixed though :) 2) Pretty slow. The entities are transfered over the network using RMI. For transferring 5000 product records ( along with their relations - product_price, product_assoc, etc ) it took over 2 hours using pretty good hard and 50 mbits network connection. 3) The example of entities to sync in the sync howto notes will never work. There will be records with duplicate PKs :) The solution for those issues imho are: 1) Refactor a bit the entity sync code so the client remembers the last successful sync state. This would fix the fail over issues. currently the server ( MCS ) remembers the sync status for each client. 2) Don't use RMI :) It would be much faster if the entities that are going to be transfered in order to get synced are serialized to XML for example and transferred over the network in a text format. RMI is easy to use but has bad performance. Just to give you an example: about 5000 products ( which is probably about 20 000 entities in total to pull ) takes 2 hours on a 50 mbits connection. What if we had a pretty huge supermarket which has 100 000 sales per day ( this means 100 000 sales orders , average 50 order_items per order, 100 000 payments, 100 000 invoices, 100 000 invoice items ). It would require an entire night to sync - if we are lucky :) 3) I can provide the complete list of tables needed to get synced for a POS solution, including fix of the issue with the duplicate PKs :) However for our current project we've used triggers which create a transaction log for push on the POS databases and a transaction log for pull on the MCS instance. Pulling those 5000 products along with all the related records takes about 20 secs on the same hardware. I've configured a crontab job which invoces the sync postgre stored procedures every 5 minutes and it works perfectly. A status table is maintained on the MCS so we can check regularly if a POS hasn't synced for let's say 1 hour - this means either network outage or some other problem. Yes, I know this solution is DB dependent but if the ofbiz sync framework issues are fixed and we use the list of tables to sync i've managed to create something useful could come out of it :) So - i hope my e-mail answers your question, if you want to know more - don't hesitate :) Cheers, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: Deyan Tsvetanov <[hidden email]>, [hidden email] Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI error ) Date: Mon, 26 Oct 2009 08:34:51 +0100 Hi Deyan, Did you work on this ? I will need to use EntitySync soon and I know you are not the 1st to complain about it (Si Chen did for instance). On the other hand, some seems to have used it successfully... Thanks Jacques ----- Original Message ----- From: Deyan Tsvetanov To: Jacques Le Roux ; [hidden email] Sent: Sunday, October 04, 2009 10:57 AM Subject: EntitySync issues & suggestions ( was EntitySync RMI error ) Hi Jacques and list, After playing for almost 2 weeks with the entity sync feature I've come up with the conclusion that it still needs some work to get ready for production environment. Currently it is unreliable, and useless :( - There is a bug in the design. The MCS remembers the sync status and currently reached sync timestamp for each client. This is not a good approach imho. Why ? The client asks the server: have you got something for me ? Next: The MCS checks the last successful sync timestamp for that particular client and selects the entities that should be passed to the client. After successfully transferring the entities to the client over RMI the MCS assumes that the sync is successful and updates the last successful sync timestamp. The problem is that the client does not confirm the success of the operation. If for some reason the client is unable to update the entities ( either shutdown, power failure, DB error or whatever ) the MCS isn't aware of that and the next sync will not include the missed entities. There is no way to fix the skipped entities but re-creating the client's database from zero. That is the main issue actually. It could be easily solved however. The most easy ways are: 1) The client will remember the sync timestamp and will include it as an argument when asking the server for pull sync. This way the client will update the last successful sync timestamp after storing each entity. The server does not have to remember anything regarding the client's sync status. 2) After each sync session the server will wait for confirmation from the client before updating the last successful sync timestamp. If not updated than the sync process is considered to be unsuccessful and the timestamp will not be updated. I personally prefer approach 1) as in case of network error, power failure or whatever error the sync process will continue from the place it has stopped. - There are some swallowed exceptions. If a network problem occurs during the sync process the server's status remains "sync running" and the client is not able to sync anymore until the status is reset by an operator. - The last issue is the performance. Using RMI is an easy approach from developer's perspective but it is very very slow. I had a database with 5000 products. The initial sync took about 2 hours. The hardware is pretty good, the network connection between the MCS and the client ofbiz is about 50 mbits. What if we have 100 000 sales per day ? The sync process will take probably the entire night :) Here I would suggest transferring GenericEntity instances over RMI to be removed. Instead a regular XML could be used. Either over RMI or SOAP - does not matter. So in general the picture would be: 1) Server does not remember any timestamp for any client. 2) Client calls the pull entity sync RMI ( or soap ) method and passes the entity sync ID and the timestamp of the last successful sync as arguments. Including the timezone of course :) 3) The server generates an XML file for all the entities in that entity sync group using the timestamp provided by the client. 4) Client starts updating the entities described in the received XML line by line. After each entity is updated successfully the timestamp is also updated ( on the client side - the server does not remember any client timestamps ). 5) If a failure occurs in the middle of the update process what we'll have is a sync process completed to like 50%. With the correct timestamp. When the next sync job starts the client will use the timestamp and the server will generate a new XML from the place the sync has failed. Of course an optimization could be used - the XML will be stored in a new db table so the client can continue storing the entities without asking the server to generate something he has already done. Some improvements that could be also useful: 1) The client ofbiz instances have sequence-id-prefix. So when the server is accepting push syncs it could also check the prefixes of the PKs and disallow entities without the correct prefix in the primary key. That's just a cheap insurance against errors :) We don't want store A to make sales on behalf of store B just because somebody made an error while configuring the pos ofbiz instance. 2) A time synchronization mechanism could be also implemented. In general when syncing there is a requirement that the server's and client's clocks are in sync ( ntp ). We don't want to set the system clock in ofbiz, but we could implement a check if the clocks are in sync and refuse to start the synchronization in certain cases. 3) A new column could be added to all the entities: isFromEntitySync : flag. This way we could avoid cases in which the client is trying to push records to the server which were actually pulled by the server. Yes, I know the entity's created timestamp should help us avoid such cases, but when the sync process is interrupted things like that could happen. Otherwise for our current project we've implemented a DB level synchronization - using triggers for INSERT, UPDATE and DELETE and a stored procedure which syncs the clients with a MCS. It works much ( tens of times ) faster than ofbiz entity sync . But it is DB dependent. So after getting comments and suggestions to this e-mail I would like to invest some time in fixing all those issues with the ofbiz entity sync and switch to it when it's ready :) Thanks for your time, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
Administrator
|
Thanks for comment Deyan,
I will try Pankaj Jain's tips in a 1st time Jacques From: "Deyan Tsvetanov" <[hidden email]> Hi Jacques, no, I didn't work on fixing the entity sync issues. Instead I did my own sync tool using db triggers and stored procedures. The entity sync could be fixed, but it would require like two - three weeks of work. I am sorry I didn't post any details, i was kind of busy because of a missed deadline :) The entity sync is kind of useless in this state imho. Reasons: 1) Does not do any fail over - if the sync process is interrupted for any reason - like network outage, or ofbiz restart, or system freeze it will be considered as a success sync and won't continue from the point it has failed. Could be fixed though :) 2) Pretty slow. The entities are transfered over the network using RMI. For transferring 5000 product records ( along with their relations - product_price, product_assoc, etc ) it took over 2 hours using pretty good hard and 50 mbits network connection. 3) The example of entities to sync in the sync howto notes will never work. There will be records with duplicate PKs :) The solution for those issues imho are: 1) Refactor a bit the entity sync code so the client remembers the last successful sync state. This would fix the fail over issues. currently the server ( MCS ) remembers the sync status for each client. 2) Don't use RMI :) It would be much faster if the entities that are going to be transfered in order to get synced are serialized to XML for example and transferred over the network in a text format. RMI is easy to use but has bad performance. Just to give you an example: about 5000 products ( which is probably about 20 000 entities in total to pull ) takes 2 hours on a 50 mbits connection. What if we had a pretty huge supermarket which has 100 000 sales per day ( this means 100 000 sales orders , average 50 order_items per order, 100 000 payments, 100 000 invoices, 100 000 invoice items ). It would require an entire night to sync - if we are lucky :) 3) I can provide the complete list of tables needed to get synced for a POS solution, including fix of the issue with the duplicate PKs :) However for our current project we've used triggers which create a transaction log for push on the POS databases and a transaction log for pull on the MCS instance. Pulling those 5000 products along with all the related records takes about 20 secs on the same hardware. I've configured a crontab job which invoces the sync postgre stored procedures every 5 minutes and it works perfectly. A status table is maintained on the MCS so we can check regularly if a POS hasn't synced for let's say 1 hour - this means either network outage or some other problem. Yes, I know this solution is DB dependent but if the ofbiz sync framework issues are fixed and we use the list of tables to sync i've managed to create something useful could come out of it :) So - i hope my e-mail answers your question, if you want to know more - don't hesitate :) Cheers, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: Deyan Tsvetanov <[hidden email]>, [hidden email] Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI error ) Date: Mon, 26 Oct 2009 08:34:51 +0100 Hi Deyan, Did you work on this ? I will need to use EntitySync soon and I know you are not the 1st to complain about it (Si Chen did for instance). On the other hand, some seems to have used it successfully... Thanks Jacques ----- Original Message ----- From: Deyan Tsvetanov To: Jacques Le Roux ; [hidden email] Sent: Sunday, October 04, 2009 10:57 AM Subject: EntitySync issues & suggestions ( was EntitySync RMI error ) Hi Jacques and list, After playing for almost 2 weeks with the entity sync feature I've come up with the conclusion that it still needs some work to get ready for production environment. Currently it is unreliable, and useless :( - There is a bug in the design. The MCS remembers the sync status and currently reached sync timestamp for each client. This is not a good approach imho. Why ? The client asks the server: have you got something for me ? Next: The MCS checks the last successful sync timestamp for that particular client and selects the entities that should be passed to the client. After successfully transferring the entities to the client over RMI the MCS assumes that the sync is successful and updates the last successful sync timestamp. The problem is that the client does not confirm the success of the operation. If for some reason the client is unable to update the entities ( either shutdown, power failure, DB error or whatever ) the MCS isn't aware of that and the next sync will not include the missed entities. There is no way to fix the skipped entities but re-creating the client's database from zero. That is the main issue actually. It could be easily solved however. The most easy ways are: 1) The client will remember the sync timestamp and will include it as an argument when asking the server for pull sync. This way the client will update the last successful sync timestamp after storing each entity. The server does not have to remember anything regarding the client's sync status. 2) After each sync session the server will wait for confirmation from the client before updating the last successful sync timestamp. If not updated than the sync process is considered to be unsuccessful and the timestamp will not be updated. I personally prefer approach 1) as in case of network error, power failure or whatever error the sync process will continue from the place it has stopped. - There are some swallowed exceptions. If a network problem occurs during the sync process the server's status remains "sync running" and the client is not able to sync anymore until the status is reset by an operator. - The last issue is the performance. Using RMI is an easy approach from developer's perspective but it is very very slow. I had a database with 5000 products. The initial sync took about 2 hours. The hardware is pretty good, the network connection between the MCS and the client ofbiz is about 50 mbits. What if we have 100 000 sales per day ? The sync process will take probably the entire night :) Here I would suggest transferring GenericEntity instances over RMI to be removed. Instead a regular XML could be used. Either over RMI or SOAP - does not matter. So in general the picture would be: 1) Server does not remember any timestamp for any client. 2) Client calls the pull entity sync RMI ( or soap ) method and passes the entity sync ID and the timestamp of the last successful sync as arguments. Including the timezone of course :) 3) The server generates an XML file for all the entities in that entity sync group using the timestamp provided by the client. 4) Client starts updating the entities described in the received XML line by line. After each entity is updated successfully the timestamp is also updated ( on the client side - the server does not remember any client timestamps ). 5) If a failure occurs in the middle of the update process what we'll have is a sync process completed to like 50%. With the correct timestamp. When the next sync job starts the client will use the timestamp and the server will generate a new XML from the place the sync has failed. Of course an optimization could be used - the XML will be stored in a new db table so the client can continue storing the entities without asking the server to generate something he has already done. Some improvements that could be also useful: 1) The client ofbiz instances have sequence-id-prefix. So when the server is accepting push syncs it could also check the prefixes of the PKs and disallow entities without the correct prefix in the primary key. That's just a cheap insurance against errors :) We don't want store A to make sales on behalf of store B just because somebody made an error while configuring the pos ofbiz instance. 2) A time synchronization mechanism could be also implemented. In general when syncing there is a requirement that the server's and client's clocks are in sync ( ntp ). We don't want to set the system clock in ofbiz, but we could implement a check if the clocks are in sync and refuse to start the synchronization in certain cases. 3) A new column could be added to all the entities: isFromEntitySync : flag. This way we could avoid cases in which the client is trying to push records to the server which were actually pulled by the server. Yes, I know the entity's created timestamp should help us avoid such cases, but when the sync process is interrupted things like that could happen. Otherwise for our current project we've implemented a DB level synchronization - using triggers for INSERT, UPDATE and DELETE and a stored procedure which syncs the clients with a MCS. It works much ( tens of times ) faster than ofbiz entity sync . But it is DB dependent. So after getting comments and suggestions to this e-mail I would like to invest some time in fixing all those issues with the ofbiz entity sync and switch to it when it's ready :) Thanks for your time, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
Administrator
|
Hi Deyan, Pankaj,
Could you share your work done so far with the community ? I agree that we shouyd try to use and maintain EntitySync. But for the volume reason Deyan outlined and in the meantime I'm sure his solution would be appreciated too! I suggest to create Jira or/and wiki pages. I'm ready to help on this. Thanks Jacques From: "Jacques Le Roux" <[hidden email]> > Thanks for comment Deyan, > > I will try Pankaj Jain's tips in a 1st time > > Jacques > > From: "Deyan Tsvetanov" <[hidden email]> > Hi Jacques, > > no, I didn't work on fixing the entity sync issues. > Instead I did my own sync tool using db triggers and stored procedures. > > The entity sync could be fixed, but it would require like two - three > weeks of work. > I am sorry I didn't post any details, i was kind of busy because of a > missed deadline :) > > The entity sync is kind of useless in this state imho. > Reasons: > > 1) Does not do any fail over - if the sync process is interrupted for > any reason - like network outage, or ofbiz restart, or system freeze > it will be considered as a success sync and won't continue from the > point it has failed. Could be fixed though :) > > 2) Pretty slow. The entities are transfered over the network using RMI. > For transferring 5000 product records ( along with their relations - > product_price, product_assoc, etc ) > it took over 2 hours using pretty good hard and 50 mbits network > connection. > > 3) The example of entities to sync in the sync howto notes will never > work. There will be records with duplicate PKs :) > > The solution for those issues imho are: > 1) Refactor a bit the entity sync code so the client remembers the last > successful sync state. This would fix the fail over issues. currently > the server ( MCS ) remembers the > sync status for each client. > > 2) Don't use RMI :) It would be much faster if the entities that are > going to be transfered in order to get synced are serialized to XML for > example and > transferred over the network in a text format. RMI is easy to use but > has bad performance. Just to give you an example: > about 5000 products ( which is probably about 20 000 entities in total > to pull ) takes 2 hours on a 50 mbits connection. What if we had a > pretty huge supermarket > which has 100 000 sales per day ( this means 100 000 sales orders , > average 50 order_items per order, 100 000 payments, 100 000 invoices, > 100 000 invoice items ). > It would require an entire night to sync - if we are lucky :) > > 3) I can provide the complete list of tables needed to get synced for a > POS solution, including fix of the issue with the duplicate PKs :) > > However for our current project we've used triggers which create a > transaction log for push on the POS databases and a transaction log for > pull on the MCS instance. > Pulling those 5000 products along with all the related records takes > about 20 secs on the same hardware. I've configured a crontab job which > invoces the sync postgre stored procedures > every 5 minutes and it works perfectly. A status table is maintained on > the MCS so we can check regularly if a POS hasn't synced for let's say 1 > hour - this means either network outage or some other problem. > > Yes, I know this solution is DB dependent but if the ofbiz sync > framework issues are fixed and we use the list of tables to sync i've > managed to create something useful could come out of it :) > > So - i hope my e-mail answers your question, if you want to know more - > don't hesitate :) > > Cheers, > Deyan > > -----Original Message----- > From: Jacques Le Roux <[hidden email]> > Reply-to: "Jacques Le Roux" <[hidden email]> > To: Deyan Tsvetanov <[hidden email]>, > [hidden email] > Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI > error ) > Date: Mon, 26 Oct 2009 08:34:51 +0100 > > > Hi Deyan, > > Did you work on this ? I will need to use EntitySync soon and I know you > are not the 1st to complain about it (Si Chen did for instance). On the > other hand, some seems to have used it successfully... > > Thanks > > Jacques > ----- Original Message ----- > From: Deyan Tsvetanov > To: Jacques Le Roux ; [hidden email] > Sent: Sunday, October 04, 2009 10:57 AM > Subject: EntitySync issues & suggestions ( was EntitySync RMI > error ) > > > > Hi Jacques and list, > > After playing for almost 2 weeks with the entity sync feature > I've come up with the conclusion that it still needs > some work to get ready for production environment. Currently it > is unreliable, and useless :( > > - There is a bug in the design. The MCS remembers the sync > status and currently reached sync timestamp for each client. > This is not a good approach imho. Why ? > The client asks the server: have you got something for me ? > Next: The MCS checks the last successful sync timestamp for that > particular client and selects the entities that should be passed > to the client. After successfully transferring the entities to > the client over RMI the MCS assumes that the sync is successful > and updates the last successful sync timestamp. > > The problem is that the client does not confirm the success of > the operation. If for some reason the client is unable to update > the entities ( either shutdown, > power failure, DB error or whatever ) the MCS isn't aware of > that and the next sync will not include the missed entities. > There is no way to fix the skipped > entities but re-creating the client's database from zero. > > That is the main issue actually. It could be easily solved > however. The most easy ways are: > 1) The client will remember the sync timestamp and will include > it as an argument when asking the server for pull sync. This way > the client will > update the last successful sync timestamp after storing each > entity. The server does not have to remember anything regarding > the client's sync status. > > 2) After each sync session the server will wait for confirmation > from the client before updating the last successful sync > timestamp. > If not updated than the sync process is considered to be > unsuccessful and the timestamp will not be updated. > > I personally prefer approach 1) as in case of network error, > power failure or whatever error the sync process will continue > from the place it has stopped. > > > - There are some swallowed exceptions. If a network problem > occurs during the sync process the server's status remains "sync > running" and the client is > not able to sync anymore until the status is reset by an > operator. > > - The last issue is the performance. Using RMI is an easy > approach from developer's perspective but it is very very slow. > I had a database with 5000 products. The initial sync took about > 2 hours. The hardware is pretty good, the network connection > between the MCS and the client ofbiz > is about 50 mbits. What if we have 100 000 sales per day ? The > sync process will take probably the entire night :) > Here I would suggest transferring GenericEntity instances over > RMI to be removed. Instead a regular XML could be used. Either > over RMI or SOAP - does not matter. > > So in general the picture would be: > > 1) Server does not remember any timestamp for any client. > 2) Client calls the pull entity sync RMI ( or soap ) method and > passes the entity sync ID and the timestamp of the last > successful sync as arguments. Including the timezone of > course :) > 3) The server generates an XML file for all the entities in that > entity sync group using the timestamp provided by the client. > 4) Client starts updating the entities described in the received > XML line by line. After each entity is updated successfully the > timestamp is also updated ( on the client side - the > server does not remember any client timestamps ). > 5) If a failure occurs in the middle of the update process what > we'll have is a sync process completed to like 50%. With the > correct timestamp. When the next sync > job starts the client will use the timestamp and the server will > generate a new XML from the place the sync has failed. Of course > an optimization could be used - > the XML will be stored in a new db table so the client can > continue storing the entities without asking the server to > generate something he has already done. > > Some improvements that could be also useful: > 1) The client ofbiz instances have sequence-id-prefix. So when > the server is accepting push syncs it could also check the > prefixes of the PKs and disallow entities without the correct > prefix in the primary key. That's just a cheap insurance against > errors :) We don't want store A to make sales on behalf of store > B just because somebody made an error while configuring the > pos ofbiz instance. > > 2) A time synchronization mechanism could be also implemented. > In general when syncing there is a requirement that the server's > and client's clocks are in sync ( ntp ). We don't want > to set the system clock in ofbiz, but we could implement a check > if the clocks are in sync and refuse to start the > synchronization in certain cases. > > 3) A new column could be added to all the entities: > isFromEntitySync : flag. This way we could avoid cases in which > the client is trying to push records to the server which were > actually pulled by the server. Yes, I know the entity's created > timestamp should help us avoid such cases, but when the sync > process is interrupted things like that could happen. > > Otherwise for our current project we've implemented a DB level > synchronization - using triggers for INSERT, UPDATE and DELETE > and a stored procedure which syncs the clients with a MCS. > It works much ( tens of times ) faster than ofbiz entity sync . > But it is DB dependent. So after getting comments and > suggestions to this e-mail I would like to invest some time in > fixing all those issues with the ofbiz entity sync and switch to > it when it's ready :) > > Thanks for your time, > Deyan > > -----Original Message----- > From: Jacques Le Roux <[hidden email]> > Reply-to: "Jacques Le Roux" <[hidden email]> > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sun, 27 Sep 2009 17:02:57 +0200 > > > Hi Deyan, > > It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to > put it, I > will eventually take care to put it under FAQ... > > TIA > > Jacques > > From: "Deyan Tsvetanov" <[hidden email]> > >I found an issue though: > > > > If the connection gets dropped during sync : > > - the client ( POS ) prints connection reset by peer > > - the server ( MCS ) sync status remains to running. > > - the next sync does not start because MCS complains there is another > > sync running already. > > > > "Reset run status" from webtools -> entity sync status > > helps. > > > > > > I'll investigate further and log a bug, although the solution seems > > pretty simple - looks like a swallowed IOException, > > which should be handled by re-setting the sync status. > > > > CHeers, > > DEyan > > > > -----Original Message----- > > From: Deyan Tsvetanov <[hidden email]> > > Reply-to: [hidden email] > > To: [hidden email] > > Subject: Re: EntitySync RMI error > > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > > > > Got it ! > > > > Seems that the previous exception was coming from the MCS ( server ) . > > When I stopped the server I started getting the exception I kind of like > > more :) > > > > > > Message: Exception calling remote pull and report EntitySync service > > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > > ricServiceException: RMI Error (Connection refused to host: > > 192.168.1.100; nested exception is: > > java.net.ConnectException: Connection refused: connect) > > > > So the problem was at the server side - RMIIF env var in startofbiz.sh > > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > > hostname . > > > > The misleading things for me was that: > > 1) no any sign in the MCS logs that somebody is trying to connect and > > the MCS itself can not connect to its own RMI registry. > > 2) no any difference in the exceptions text on the POS ( client ) side: > > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > > remote one - can not connect to 127.0.0.1 ) > > look the same :) > > > > So issue is solved, > > sorry for bothering :) > > > > -- deyan > > > > -----Original Message----- > > From: Deyan Tsvetanov <[hidden email]> > > Reply-to: [hidden email] > > To: [hidden email] > > Subject: Re: EntitySync RMI error > > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > > > > During startup I get: > > > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > > [main-rmi=rmi://127.0.0. > > 1:1099/RMIDispatcher, > > main-http=http://127.0.0.1:8080/webtools/control/httpService, > > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > > 0.0.1:8080/webtools/control/httpService] > > > > entity-sync-rmi seems to be ok ... > > > > > > > > > > -----Original Message----- > > From: Deyan Tsvetanov <[hidden email]> > > Reply-to: [hidden email] > > To: [hidden email] > > Subject: EntitySync RMI error > > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > > > > Hi guys, > > > > I'm trying to configure RMI entity sync. I'm following > > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > > > What I've done so far: > > > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > > ( as per the example ). > > > > 3) I've imported the entity sync groups, SandJobs, etc. > > > > However when the sync starts ( on the POS instance ) I get the following > > error: > > > > Exception calling remote pull and report EntitySync service with name: > > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > > nested exception is: > > java.net.ConnectException: Connection refused: connect) > > Exception: org.ofbiz.service.GenericServiceException > > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > > nested exception is: > > java.net.ConnectException: Connection refused: connect) > > ---- cause > > --------------------------------------------------------------------- > > Exception: java.rmi.ConnectException > > Message: Connection refused to host: 127.0.0.1; nested exception is: > > java.net.ConnectException: Connection refused: connect > > ---- cause > > --------------------------------------------------------------------- > > Exception: java.net.ConnectException > > Message: Connection refused: connect > > ---- stack trace > > --------------------------------------------------------------- > > > > > > It insists connecting to 127.0.0.1 no matter what I type in > > serviceengine.xml. > > Any help would be appreciated :) > > > > Thanks in advance, > > Deyan > > > > > > |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
I have a customer who is also interested...
Hans On Tue, 2009-10-27 at 08:58 +0100, Jacques Le Roux wrote: > Hi Deyan, Pankaj, > > Could you share your work done so far with the community ? > I agree that we shouyd try to use and maintain EntitySync. > But for the volume reason Deyan outlined and in the meantime I'm sure his solution would be appreciated too! > I suggest to create Jira or/and wiki pages. I'm ready to help on this. > > Thanks > > Jacques > > From: "Jacques Le Roux" <[hidden email]> > > Thanks for comment Deyan, > > > > I will try Pankaj Jain's tips in a 1st time > > > > Jacques > > > > From: "Deyan Tsvetanov" <[hidden email]> > > Hi Jacques, > > > > no, I didn't work on fixing the entity sync issues. > > Instead I did my own sync tool using db triggers and stored procedures. > > > > The entity sync could be fixed, but it would require like two - three > > weeks of work. > > I am sorry I didn't post any details, i was kind of busy because of a > > missed deadline :) > > > > The entity sync is kind of useless in this state imho. > > Reasons: > > > > 1) Does not do any fail over - if the sync process is interrupted for > > any reason - like network outage, or ofbiz restart, or system freeze > > it will be considered as a success sync and won't continue from the > > point it has failed. Could be fixed though :) > > > > 2) Pretty slow. The entities are transfered over the network using RMI. > > For transferring 5000 product records ( along with their relations - > > product_price, product_assoc, etc ) > > it took over 2 hours using pretty good hard and 50 mbits network > > connection. > > > > 3) The example of entities to sync in the sync howto notes will never > > work. There will be records with duplicate PKs :) > > > > The solution for those issues imho are: > > 1) Refactor a bit the entity sync code so the client remembers the last > > successful sync state. This would fix the fail over issues. currently > > the server ( MCS ) remembers the > > sync status for each client. > > > > 2) Don't use RMI :) It would be much faster if the entities that are > > going to be transfered in order to get synced are serialized to XML for > > example and > > transferred over the network in a text format. RMI is easy to use but > > has bad performance. Just to give you an example: > > about 5000 products ( which is probably about 20 000 entities in total > > to pull ) takes 2 hours on a 50 mbits connection. What if we had a > > pretty huge supermarket > > which has 100 000 sales per day ( this means 100 000 sales orders , > > average 50 order_items per order, 100 000 payments, 100 000 invoices, > > 100 000 invoice items ). > > It would require an entire night to sync - if we are lucky :) > > > > 3) I can provide the complete list of tables needed to get synced for a > > POS solution, including fix of the issue with the duplicate PKs :) > > > > However for our current project we've used triggers which create a > > transaction log for push on the POS databases and a transaction log for > > pull on the MCS instance. > > Pulling those 5000 products along with all the related records takes > > about 20 secs on the same hardware. I've configured a crontab job which > > invoces the sync postgre stored procedures > > every 5 minutes and it works perfectly. A status table is maintained on > > the MCS so we can check regularly if a POS hasn't synced for let's say 1 > > hour - this means either network outage or some other problem. > > > > Yes, I know this solution is DB dependent but if the ofbiz sync > > framework issues are fixed and we use the list of tables to sync i've > > managed to create something useful could come out of it :) > > > > So - i hope my e-mail answers your question, if you want to know more - > > don't hesitate :) > > > > Cheers, > > Deyan > > > > -----Original Message----- > > From: Jacques Le Roux <[hidden email]> > > Reply-to: "Jacques Le Roux" <[hidden email]> > > To: Deyan Tsvetanov <[hidden email]>, > > [hidden email] > > Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI > > error ) > > Date: Mon, 26 Oct 2009 08:34:51 +0100 > > > > > > Hi Deyan, > > > > Did you work on this ? I will need to use EntitySync soon and I know you > > are not the 1st to complain about it (Si Chen did for instance). On the > > other hand, some seems to have used it successfully... > > > > Thanks > > > > Jacques > > ----- Original Message ----- > > From: Deyan Tsvetanov > > To: Jacques Le Roux ; [hidden email] > > Sent: Sunday, October 04, 2009 10:57 AM > > Subject: EntitySync issues & suggestions ( was EntitySync RMI > > error ) > > > > > > > > Hi Jacques and list, > > > > After playing for almost 2 weeks with the entity sync feature > > I've come up with the conclusion that it still needs > > some work to get ready for production environment. Currently it > > is unreliable, and useless :( > > > > - There is a bug in the design. The MCS remembers the sync > > status and currently reached sync timestamp for each client. > > This is not a good approach imho. Why ? > > The client asks the server: have you got something for me ? > > Next: The MCS checks the last successful sync timestamp for that > > particular client and selects the entities that should be passed > > to the client. After successfully transferring the entities to > > the client over RMI the MCS assumes that the sync is successful > > and updates the last successful sync timestamp. > > > > The problem is that the client does not confirm the success of > > the operation. If for some reason the client is unable to update > > the entities ( either shutdown, > > power failure, DB error or whatever ) the MCS isn't aware of > > that and the next sync will not include the missed entities. > > There is no way to fix the skipped > > entities but re-creating the client's database from zero. > > > > That is the main issue actually. It could be easily solved > > however. The most easy ways are: > > 1) The client will remember the sync timestamp and will include > > it as an argument when asking the server for pull sync. This way > > the client will > > update the last successful sync timestamp after storing each > > entity. The server does not have to remember anything regarding > > the client's sync status. > > > > 2) After each sync session the server will wait for confirmation > > from the client before updating the last successful sync > > timestamp. > > If not updated than the sync process is considered to be > > unsuccessful and the timestamp will not be updated. > > > > I personally prefer approach 1) as in case of network error, > > power failure or whatever error the sync process will continue > > from the place it has stopped. > > > > > > - There are some swallowed exceptions. If a network problem > > occurs during the sync process the server's status remains "sync > > running" and the client is > > not able to sync anymore until the status is reset by an > > operator. > > > > - The last issue is the performance. Using RMI is an easy > > approach from developer's perspective but it is very very slow. > > I had a database with 5000 products. The initial sync took about > > 2 hours. The hardware is pretty good, the network connection > > between the MCS and the client ofbiz > > is about 50 mbits. What if we have 100 000 sales per day ? The > > sync process will take probably the entire night :) > > Here I would suggest transferring GenericEntity instances over > > RMI to be removed. Instead a regular XML could be used. Either > > over RMI or SOAP - does not matter. > > > > So in general the picture would be: > > > > 1) Server does not remember any timestamp for any client. > > 2) Client calls the pull entity sync RMI ( or soap ) method and > > passes the entity sync ID and the timestamp of the last > > successful sync as arguments. Including the timezone of > > course :) > > 3) The server generates an XML file for all the entities in that > > entity sync group using the timestamp provided by the client. > > 4) Client starts updating the entities described in the received > > XML line by line. After each entity is updated successfully the > > timestamp is also updated ( on the client side - the > > server does not remember any client timestamps ). > > 5) If a failure occurs in the middle of the update process what > > we'll have is a sync process completed to like 50%. With the > > correct timestamp. When the next sync > > job starts the client will use the timestamp and the server will > > generate a new XML from the place the sync has failed. Of course > > an optimization could be used - > > the XML will be stored in a new db table so the client can > > continue storing the entities without asking the server to > > generate something he has already done. > > > > Some improvements that could be also useful: > > 1) The client ofbiz instances have sequence-id-prefix. So when > > the server is accepting push syncs it could also check the > > prefixes of the PKs and disallow entities without the correct > > prefix in the primary key. That's just a cheap insurance against > > errors :) We don't want store A to make sales on behalf of store > > B just because somebody made an error while configuring the > > pos ofbiz instance. > > > > 2) A time synchronization mechanism could be also implemented. > > In general when syncing there is a requirement that the server's > > and client's clocks are in sync ( ntp ). We don't want > > to set the system clock in ofbiz, but we could implement a check > > if the clocks are in sync and refuse to start the > > synchronization in certain cases. > > > > 3) A new column could be added to all the entities: > > isFromEntitySync : flag. This way we could avoid cases in which > > the client is trying to push records to the server which were > > actually pulled by the server. Yes, I know the entity's created > > timestamp should help us avoid such cases, but when the sync > > process is interrupted things like that could happen. > > > > Otherwise for our current project we've implemented a DB level > > synchronization - using triggers for INSERT, UPDATE and DELETE > > and a stored procedure which syncs the clients with a MCS. > > It works much ( tens of times ) faster than ofbiz entity sync . > > But it is DB dependent. So after getting comments and > > suggestions to this e-mail I would like to invest some time in > > fixing all those issues with the ofbiz entity sync and switch to > > it when it's ready :) > > > > Thanks for your time, > > Deyan > > > > -----Original Message----- > > From: Jacques Le Roux <[hidden email]> > > Reply-to: "Jacques Le Roux" <[hidden email]> > > To: [hidden email] > > Subject: Re: EntitySync RMI error > > Date: Sun, 27 Sep 2009 17:02:57 +0200 > > > > > > Hi Deyan, > > > > It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to > > put it, I > > will eventually take care to put it under FAQ... > > > > TIA > > > > Jacques > > > > From: "Deyan Tsvetanov" <[hidden email]> > > >I found an issue though: > > > > > > If the connection gets dropped during sync : > > > - the client ( POS ) prints connection reset by peer > > > - the server ( MCS ) sync status remains to running. > > > - the next sync does not start because MCS complains there is another > > > sync running already. > > > > > > "Reset run status" from webtools -> entity sync status > > > helps. > > > > > > > > > I'll investigate further and log a bug, although the solution seems > > > pretty simple - looks like a swallowed IOException, > > > which should be handled by re-setting the sync status. > > > > > > CHeers, > > > DEyan > > > > > > -----Original Message----- > > > From: Deyan Tsvetanov <[hidden email]> > > > Reply-to: [hidden email] > > > To: [hidden email] > > > Subject: Re: EntitySync RMI error > > > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > > > > > > > Got it ! > > > > > > Seems that the previous exception was coming from the MCS ( server ) . > > > When I stopped the server I started getting the exception I kind of like > > > more :) > > > > > > > > > Message: Exception calling remote pull and report EntitySync service > > > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > > > ricServiceException: RMI Error (Connection refused to host: > > > 192.168.1.100; nested exception is: > > > java.net.ConnectException: Connection refused: connect) > > > > > > So the problem was at the server side - RMIIF env var in startofbiz.sh > > > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > > > hostname . > > > > > > The misleading things for me was that: > > > 1) no any sign in the MCS logs that somebody is trying to connect and > > > the MCS itself can not connect to its own RMI registry. > > > 2) no any difference in the exceptions text on the POS ( client ) side: > > > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > > > remote one - can not connect to 127.0.0.1 ) > > > look the same :) > > > > > > So issue is solved, > > > sorry for bothering :) > > > > > > -- deyan > > > > > > -----Original Message----- > > > From: Deyan Tsvetanov <[hidden email]> > > > Reply-to: [hidden email] > > > To: [hidden email] > > > Subject: Re: EntitySync RMI error > > > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > > > > > > > During startup I get: > > > > > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > > > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > > > [main-rmi=rmi://127.0.0. > > > 1:1099/RMIDispatcher, > > > main-http=http://127.0.0.1:8080/webtools/control/httpService, > > > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > > > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > > > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > > > 0.0.1:8080/webtools/control/httpService] > > > > > > entity-sync-rmi seems to be ok ... > > > > > > > > > > > > > > > -----Original Message----- > > > From: Deyan Tsvetanov <[hidden email]> > > > Reply-to: [hidden email] > > > To: [hidden email] > > > Subject: EntitySync RMI error > > > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > > > > > > > Hi guys, > > > > > > I'm trying to configure RMI entity sync. I'm following > > > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > > > > > What I've done so far: > > > > > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > > > > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > > > ( as per the example ). > > > > > > 3) I've imported the entity sync groups, SandJobs, etc. > > > > > > However when the sync starts ( on the POS instance ) I get the following > > > error: > > > > > > Exception calling remote pull and report EntitySync service with name: > > > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > > > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > > > nested exception is: > > > java.net.ConnectException: Connection refused: connect) > > > Exception: org.ofbiz.service.GenericServiceException > > > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > > > nested exception is: > > > java.net.ConnectException: Connection refused: connect) > > > ---- cause > > > --------------------------------------------------------------------- > > > Exception: java.rmi.ConnectException > > > Message: Connection refused to host: 127.0.0.1; nested exception is: > > > java.net.ConnectException: Connection refused: connect > > > ---- cause > > > --------------------------------------------------------------------- > > > Exception: java.net.ConnectException > > > Message: Connection refused: connect > > > ---- stack trace > > > --------------------------------------------------------------- > > > > > > > > > It insists connecting to 127.0.0.1 no matter what I type in > > > serviceengine.xml. > > > Any help would be appreciated :) > > > > > > Thanks in advance, > > > Deyan > > > > > > > > > > > > > Antwebsystems.com: Quality OFBiz services for competitive rates |
Re: EntitySync issues & suggestions ( was EntitySync RMI error )
|
|
In reply to this post by Jacques Le Roux
I don't use entity sync, and I haven't looked at the code, but based on
what I've read in the mailing lists, I have some ideas. Take them for what they are worth. If the problems mentioned in this thread and others really do exist, and if there is a consensus that the entity sync code needs to be reworked, then why not find an alternative to RMI - since there seems to be a performance problem there? It seems to me we could develop an XML file format that describes the update tasks needed, and another XML file format that describes the final outcome of each update task. The XML files could be zipped and transferred using regular file transfer protocols. It would be similar to how WebDAV works. Maybe we could even create a WebDAV interface for entity synch. -Adrian Jacques Le Roux wrote: > Hi Deyan, Pankaj, > > Could you share your work done so far with the community ? > I agree that we shouyd try to use and maintain EntitySync. > But for the volume reason Deyan outlined and in the meantime I'm sure > his solution would be appreciated too! > I suggest to create Jira or/and wiki pages. I'm ready to help on this. > > Thanks > > Jacques > > From: "Jacques Le Roux" <[hidden email]> >> Thanks for comment Deyan, >> >> I will try Pankaj Jain's tips in a 1st time >> >> Jacques >> >> From: "Deyan Tsvetanov" <[hidden email]> >> Hi Jacques, >> >> no, I didn't work on fixing the entity sync issues. >> Instead I did my own sync tool using db triggers and stored procedures. >> >> The entity sync could be fixed, but it would require like two - three >> weeks of work. >> I am sorry I didn't post any details, i was kind of busy because of a >> missed deadline :) >> >> The entity sync is kind of useless in this state imho. >> Reasons: >> >> 1) Does not do any fail over - if the sync process is interrupted for >> any reason - like network outage, or ofbiz restart, or system freeze >> it will be considered as a success sync and won't continue from the >> point it has failed. Could be fixed though :) >> >> 2) Pretty slow. The entities are transfered over the network using RMI. >> For transferring 5000 product records ( along with their relations - >> product_price, product_assoc, etc ) >> it took over 2 hours using pretty good hard and 50 mbits network >> connection. >> >> 3) The example of entities to sync in the sync howto notes will never >> work. There will be records with duplicate PKs :) >> >> The solution for those issues imho are: >> 1) Refactor a bit the entity sync code so the client remembers the last >> successful sync state. This would fix the fail over issues. currently >> the server ( MCS ) remembers the >> sync status for each client. >> >> 2) Don't use RMI :) It would be much faster if the entities that are >> going to be transfered in order to get synced are serialized to XML for >> example and >> transferred over the network in a text format. RMI is easy to use but >> has bad performance. Just to give you an example: >> about 5000 products ( which is probably about 20 000 entities in total >> to pull ) takes 2 hours on a 50 mbits connection. What if we had a >> pretty huge supermarket >> which has 100 000 sales per day ( this means 100 000 sales orders , >> average 50 order_items per order, 100 000 payments, 100 000 invoices, >> 100 000 invoice items ). >> It would require an entire night to sync - if we are lucky :) >> >> 3) I can provide the complete list of tables needed to get synced for a >> POS solution, including fix of the issue with the duplicate PKs :) >> >> However for our current project we've used triggers which create a >> transaction log for push on the POS databases and a transaction log for >> pull on the MCS instance. >> Pulling those 5000 products along with all the related records takes >> about 20 secs on the same hardware. I've configured a crontab job which >> invoces the sync postgre stored procedures >> every 5 minutes and it works perfectly. A status table is maintained on >> the MCS so we can check regularly if a POS hasn't synced for let's say 1 >> hour - this means either network outage or some other problem. >> >> Yes, I know this solution is DB dependent but if the ofbiz sync >> framework issues are fixed and we use the list of tables to sync i've >> managed to create something useful could come out of it :) >> >> So - i hope my e-mail answers your question, if you want to know more - >> don't hesitate :) >> >> Cheers, >> Deyan >> >> -----Original Message----- >> From: Jacques Le Roux <[hidden email]> >> Reply-to: "Jacques Le Roux" <[hidden email]> >> To: Deyan Tsvetanov <[hidden email]>, >> [hidden email] >> Subject: Re: EntitySync issues & suggestions ( was EntitySync RMI >> error ) >> Date: Mon, 26 Oct 2009 08:34:51 +0100 >> >> >> Hi Deyan, >> >> Did you work on this ? I will need to use EntitySync soon and I know you >> are not the 1st to complain about it (Si Chen did for instance). On the >> other hand, some seems to have used it successfully... >> >> Thanks >> >> Jacques >> ----- Original Message ----- From: Deyan Tsvetanov >> To: Jacques Le Roux ; [hidden email] >> Sent: Sunday, October 04, 2009 10:57 AM >> Subject: EntitySync issues & suggestions ( was EntitySync RMI >> error ) >> >> >> >> Hi Jacques and list, >> >> After playing for almost 2 weeks with the entity sync feature >> I've come up with the conclusion that it still needs >> some work to get ready for production environment. Currently it >> is unreliable, and useless :( >> >> - There is a bug in the design. The MCS remembers the sync >> status and currently reached sync timestamp for each client. >> This is not a good approach imho. Why ? >> The client asks the server: have you got something for me ? >> Next: The MCS checks the last successful sync timestamp for that >> particular client and selects the entities that should be passed >> to the client. After successfully transferring the entities to >> the client over RMI the MCS assumes that the sync is successful >> and updates the last successful sync timestamp. >> >> The problem is that the client does not confirm the success of >> the operation. If for some reason the client is unable to update >> the entities ( either shutdown, >> power failure, DB error or whatever ) the MCS isn't aware of >> that and the next sync will not include the missed entities. >> There is no way to fix the skipped >> entities but re-creating the client's database from zero. >> >> That is the main issue actually. It could be easily solved >> however. The most easy ways are: >> 1) The client will remember the sync timestamp and will include >> it as an argument when asking the server for pull sync. This way >> the client will >> update the last successful sync timestamp after storing each >> entity. The server does not have to remember anything regarding >> the client's sync status. >> >> 2) After each sync session the server will wait for confirmation >> from the client before updating the last successful sync >> timestamp. >> If not updated than the sync process is considered to be >> unsuccessful and the timestamp will not be updated. >> >> I personally prefer approach 1) as in case of network error, >> power failure or whatever error the sync process will continue >> from the place it has stopped. >> >> >> - There are some swallowed exceptions. If a network problem >> occurs during the sync process the server's status remains "sync >> running" and the client is >> not able to sync anymore until the status is reset by an >> operator. >> >> - The last issue is the performance. Using RMI is an easy >> approach from developer's perspective but it is very very slow. >> I had a database with 5000 products. The initial sync took about >> 2 hours. The hardware is pretty good, the network connection >> between the MCS and the client ofbiz >> is about 50 mbits. What if we have 100 000 sales per day ? The >> sync process will take probably the entire night :) >> Here I would suggest transferring GenericEntity instances over >> RMI to be removed. Instead a regular XML could be used. Either >> over RMI or SOAP - does not matter. >> >> So in general the picture would be: >> >> 1) Server does not remember any timestamp for any client. >> 2) Client calls the pull entity sync RMI ( or soap ) method and >> passes the entity sync ID and the timestamp of the last >> successful sync as arguments. Including the timezone of >> course :) >> 3) The server generates an XML file for all the entities in that >> entity sync group using the timestamp provided by the client. >> 4) Client starts updating the entities described in the received >> XML line by line. After each entity is updated successfully the >> timestamp is also updated ( on the client side - the >> server does not remember any client timestamps ). >> 5) If a failure occurs in the middle of the update process what >> we'll have is a sync process completed to like 50%. With the >> correct timestamp. When the next sync >> job starts the client will use the timestamp and the server will >> generate a new XML from the place the sync has failed. Of course >> an optimization could be used - >> the XML will be stored in a new db table so the client can >> continue storing the entities without asking the server to >> generate something he has already done. >> >> Some improvements that could be also useful: >> 1) The client ofbiz instances have sequence-id-prefix. So when >> the server is accepting push syncs it could also check the >> prefixes of the PKs and disallow entities without the correct >> prefix in the primary key. That's just a cheap insurance against >> errors :) We don't want store A to make sales on behalf of store >> B just because somebody made an error while configuring the >> pos ofbiz instance. >> >> 2) A time synchronization mechanism could be also implemented. >> In general when syncing there is a requirement that the server's >> and client's clocks are in sync ( ntp ). We don't want >> to set the system clock in ofbiz, but we could implement a check >> if the clocks are in sync and refuse to start the >> synchronization in certain cases. >> >> 3) A new column could be added to all the entities: >> isFromEntitySync : flag. This way we could avoid cases in which >> the client is trying to push records to the server which were >> actually pulled by the server. Yes, I know the entity's created >> timestamp should help us avoid such cases, but when the sync >> process is interrupted things like that could happen. >> >> Otherwise for our current project we've implemented a DB level >> synchronization - using triggers for INSERT, UPDATE and DELETE >> and a stored procedure which syncs the clients with a MCS. >> It works much ( tens of times ) faster than ofbiz entity sync . >> But it is DB dependent. So after getting comments and >> suggestions to this e-mail I would like to invest some time in >> fixing all those issues with the ofbiz entity sync and switch to >> it when it's ready :) >> >> Thanks for your time, >> Deyan >> >> -----Original Message----- >> From: Jacques Le Roux <[hidden email]> >> Reply-to: "Jacques Le Roux" <[hidden email]> >> To: [hidden email] >> Subject: Re: EntitySync RMI error >> Date: Sun, 27 Sep 2009 17:02:57 +0200 >> >> >> Hi Deyan, >> >> It would be very valuable if you could post an article on wiki >> about your experience with this. Don't worry about where to put it, I >> will eventually take care to put it under FAQ... >> >> TIA >> >> Jacques >> >> From: "Deyan Tsvetanov" <[hidden email]> >> >I found an issue though: >> > >> > If the connection gets dropped during sync : >> > - the client ( POS ) prints connection reset by peer >> > - the server ( MCS ) sync status remains to running. >> > - the next sync does not start because MCS complains there is >> another >> > sync running already. >> > >> > "Reset run status" from webtools -> entity sync status >> > helps. >> > >> > >> > I'll investigate further and log a bug, although the solution >> seems >> > pretty simple - looks like a swallowed IOException, >> > which should be handled by re-setting the sync status. >> > >> > CHeers, >> > DEyan >> > >> > -----Original Message----- >> > From: Deyan Tsvetanov <[hidden email]> >> > Reply-to: [hidden email] >> > To: [hidden email] >> > Subject: Re: EntitySync RMI error >> > Date: Sat, 26 Sep 2009 11:22:55 +0300 >> > >> > >> > Got it ! >> > >> > Seems that the previous exception was coming from the MCS ( >> server ) . >> > When I stopped the server I started getting the exception I >> kind of like >> > more :) >> > >> > >> > Message: Exception calling remote pull and report EntitySync >> service >> > with name: remotePullAndReportEntitySyncDataRmi; >> org.ofbiz.service.Gene >> > ricServiceException: RMI Error (Connection refused to host: >> > 192.168.1.100; nested exception is: >> > java.net.ConnectException: Connection refused: connect) >> > >> > So the problem was at the server side - RMIIF env var in >> startofbiz.sh >> > wasn't set , so ofbiz was trying to get RMI host IP by >> resolving the >> > hostname . >> > >> > The misleading things for me was that: >> > 1) no any sign in the MCS logs that somebody is trying to >> connect and >> > the MCS itself can not connect to its own RMI registry. >> > 2) no any difference in the exceptions text on the POS ( >> client ) side: >> > the 2 exceptions ( local one - can not connect to >> 192.168.1.100 and the >> > remote one - can not connect to 127.0.0.1 ) >> > look the same :) >> > >> > So issue is solved, >> > sorry for bothering :) >> > >> > -- deyan >> > >> > -----Original Message----- >> > From: Deyan Tsvetanov <[hidden email]> >> > Reply-to: [hidden email] >> > To: [hidden email] >> > Subject: Re: EntitySync RMI error >> > Date: Sat, 26 Sep 2009 09:52:55 +0300 >> > >> > >> > During startup I get: >> > >> > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ >> > AbstractEngine.java:73 :INFO ] Loaded Service Locations : >> > [main-rmi=rmi://127.0.0. >> > 1:1099/RMIDispatcher, >> > main-http=http://127.0.0.1:8080/webtools/control/httpService, >> > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, >> > >> entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, >> > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, >> eedcc-test=http://127. >> > 0.0.1:8080/webtools/control/httpService] >> > >> > entity-sync-rmi seems to be ok ... >> > >> > >> > >> > >> > -----Original Message----- >> > From: Deyan Tsvetanov <[hidden email]> >> > Reply-to: [hidden email] >> > To: [hidden email] >> > Subject: EntitySync RMI error >> > Date: Sat, 26 Sep 2009 09:22:23 +0300 >> > >> > >> > Hi guys, >> > >> > I'm trying to configure RMI entity sync. I'm following >> > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example >> > >> > What I've done so far: >> > >> > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher >> > >> > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 >> > ( as per the example ). >> > >> > 3) I've imported the entity sync groups, SandJobs, etc. >> > >> > However when the sync starts ( on the POS instance ) I get >> the following >> > error: >> > >> > Exception calling remote pull and report EntitySync service >> with name: >> > remotePullAndReportEntitySyncDataRmi; >> org.ofbiz.service.GenericServic >> > eException: RMI Invocation Error (Connection refused to host: >> 127.0.0.1; >> > nested exception is: >> > java.net.ConnectException: Connection refused: connect) >> > Exception: org.ofbiz.service.GenericServiceException >> > Message: RMI Invocation Error (Connection refused to host: >> 127.0.0.1; >> > nested exception is: >> > java.net.ConnectException: Connection refused: connect) >> > ---- cause >> > >> --------------------------------------------------------------------- >> > Exception: java.rmi.ConnectException >> > Message: Connection refused to host: 127.0.0.1; nested >> exception is: >> > java.net.ConnectException: Connection refused: connect >> > ---- cause >> > >> --------------------------------------------------------------------- >> > Exception: java.net.ConnectException >> > Message: Connection refused: connect >> > ---- stack trace >> > --------------------------------------------------------------- >> > >> > >> > It insists connecting to 127.0.0.1 no matter what I type in >> > serviceengine.xml. >> > Any help would be appreciated :) >> > >> > Thanks in advance, >> > Deyan >> > >> >> >> >> > > > |
|
Administrator
|
In reply to this post by Deyan Tsvetanov
Not sure it's still useful, but I was curious and just had a look at this: it should work
String entityOrPackage = entitySyncInclude.getString("entityOrPackage"); boolean matches = false; if (entityName.equals(entityOrPackage)) { matches = true; } else if (modelEntity.getPackageName().startsWith(entityOrPackage)) { matches = true; } I did not try though.... Jacques ----- Original Message ----- From: Deyan Tsvetanov To: Jacques Le Roux Cc: [hidden email] Sent: Tuesday, September 29, 2009 3:03 PM Subject: Re: EntitySync RMI error Hi guys, I've a question: <EntityGroupEntry entityGroupId="5500" entityOrPackage="org.ofbiz.product" applEnumId="ESIA_INCLUDE"/> Are the sub-packages going to be included in the sync ( e.g. org.ofbiz.product.facility , org.ofbiz.product.price, etc ) or they have to be referenced implicitly ? Looks like they aren't to me ... Thanks, Deyan -----Original Message----- From: Jacques Le Roux <[hidden email]> Reply-to: "Jacques Le Roux" <[hidden email]> To: [hidden email] Subject: Re: EntitySync RMI error Date: Sun, 27 Sep 2009 17:02:57 +0200 Hi Deyan, It would be very valuable if you could post an article on wiki about your experience with this. Don't worry about where to put it, I will eventually take care to put it under FAQ... TIA Jacques From: "Deyan Tsvetanov" <[hidden email]> >I found an issue though: > > If the connection gets dropped during sync : > - the client ( POS ) prints connection reset by peer > - the server ( MCS ) sync status remains to running. > - the next sync does not start because MCS complains there is another > sync running already. > > "Reset run status" from webtools -> entity sync status > helps. > > > I'll investigate further and log a bug, although the solution seems > pretty simple - looks like a swallowed IOException, > which should be handled by re-setting the sync status. > > CHeers, > DEyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 11:22:55 +0300 > > > Got it ! > > Seems that the previous exception was coming from the MCS ( server ) . > When I stopped the server I started getting the exception I kind of like > more :) > > > Message: Exception calling remote pull and report EntitySync service > with name: remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.Gene > ricServiceException: RMI Error (Connection refused to host: > 192.168.1.100; nested exception is: > java.net.ConnectException: Connection refused: connect) > > So the problem was at the server side - RMIIF env var in startofbiz.sh > wasn't set , so ofbiz was trying to get RMI host IP by resolving the > hostname . > > The misleading things for me was that: > 1) no any sign in the MCS logs that somebody is trying to connect and > the MCS itself can not connect to its own RMI registry. > 2) no any difference in the exceptions text on the POS ( client ) side: > the 2 exceptions ( local one - can not connect to 192.168.1.100 and the > remote one - can not connect to 127.0.0.1 ) > look the same :) > > So issue is solved, > sorry for bothering :) > > -- deyan > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: Re: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:52:55 +0300 > > > During startup I get: > > 2009-09-26 08:46:31,905 (default-invoker-Thread-10) [ > AbstractEngine.java:73 :INFO ] Loaded Service Locations : > [main-rmi=rmi://127.0.0. > 1:1099/RMIDispatcher, > main-http=http://127.0.0.1:8080/webtools/control/httpService, > entity-sync-rmi=rmi://192.168.1.100:1099/RMIDispatcher, > entity-sync-http=http://192.168.1.100:8080/webtools/control/httpService, > rita-rmi=rmi://127.0.0.1:1099/RMIDispatcher, eedcc-test=http://127. > 0.0.1:8080/webtools/control/httpService] > > entity-sync-rmi seems to be ok ... > > > > > -----Original Message----- > From: Deyan Tsvetanov <[hidden email]> > Reply-to: [hidden email] > To: [hidden email] > Subject: EntitySync RMI error > Date: Sat, 26 Sep 2009 09:22:23 +0300 > > > Hi guys, > > I'm trying to configure RMI entity sync. I'm following > http://docs.ofbiz.org/display/OFBIZ/Sync+Setup+Notes+and+Example > > What I've done so far: > > 1) entity-sync-rmi to rmi://192.168.1.100:1099/RMIDispatcher > > 2) set RMIIF=-Djava.rmi.server.hostname=127.0.0.1 > ( as per the example ). > > 3) I've imported the entity sync groups, SandJobs, etc. > > However when the sync starts ( on the POS instance ) I get the following > error: > > Exception calling remote pull and report EntitySync service with name: > remotePullAndReportEntitySyncDataRmi; org.ofbiz.service.GenericServic > eException: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > Exception: org.ofbiz.service.GenericServiceException > Message: RMI Invocation Error (Connection refused to host: 127.0.0.1; > nested exception is: > java.net.ConnectException: Connection refused: connect) > ---- cause > --------------------------------------------------------------------- > Exception: java.rmi.ConnectException > Message: Connection refused to host: 127.0.0.1; nested exception is: > java.net.ConnectException: Connection refused: connect > ---- cause > --------------------------------------------------------------------- > Exception: java.net.ConnectException > Message: Connection refused: connect > ---- stack trace > --------------------------------------------------------------- > > > It insists connecting to 127.0.0.1 no matter what I type in > serviceengine.xml. > Any help would be appreciated :) > > Thanks in advance, > Deyan > |
«
Return to OFBiz - User
|
1 view|%1 views
| Free forum by Nabble | Edit this page |