Scheduled jobs weird behavior

[Giulio Speri - MpStyle Srl]

Hi all,

I write because I am facing strange behaviors with scheduled jobs and I am
running out of ideas about causes.

I have a two instances configuration of ofbiz running on two different
servers (hosted by a third party company); they handle two eCommerce site
on a multi-tenant environment.
OFBiz requests are served via an Apache Server that redirects them to one
of the two instances with a load balancer.
OFBiz version is 13.07.03.


The problem is this.

I have a service "exportOrders" the must run three times a day, in fixed
times (ex: 7, 11, 14), so I scheduled three different jobs, one for each
runtime (so job names will be something like : "ServiceName 0700",
"ServiceName 1100", "ServiceName 1400") that run the same service.
In short, the service reads ecommerce orders data, creates four text files
and zip them.
The result zip file is written in a "history" directory on the server and
then sent via ftp to a logistic company server.
To know if a certain order has been processed by this service, I added a
field "isExported" (type indicator) on OrderHeader entity, that is set to
"Y" at the end of the "exportOrders".
Within this service I also write a simple text file that logs operations
made for each execution of it (the file name is always different).

Now, here is the issue; sometimes (runtime in which happen seems random), I
notice that:
- the service in the Job List screen, has the state of "Finished" (no
errors);
- No log file nor zip file is written;
- the flag on the OrderHeader entity has been set to "Y" (by default is
null this field);
- no error or exception found in ofbiz/error log files;

The Job Poller on the second instance of ofbiz is disabled
("poll-enable=false" in serviceengine.xml), so that all the scheduled jobs
are run only on the first instance.

I am quite confused by this situation and I really don't know what and
where to look, to understand what happens.

Does anyone ever experienced a similar issue?
Thanks in advance for your help.

Giulio

--
Giulio Speri


*Mp Styl**e Srl*
via Antonio Meucci, 37
41019 Limidi di Soliera (MO)
T 059/684916
M 334/3779851 <(334)%20377-9851>

www.mpstyle.it

[Nicolas Malin-2]

Hello Giulio,

On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:

> Hi all,
> [...] Now, here is the issue; sometimes (runtime in which happen seems random), I
> notice that:
> - the service in the Job List screen, has the state of "Finished" (no
> errors);
> - No log file nor zip file is written;
> - the flag on the OrderHeader entity has been set to "Y" (by default is
> null this field);
> - no error or exception found in ofbiz/error log files;
>
> The Job Poller on the second instance of ofbiz is disabled
> ("poll-enable=false" in serviceengine.xml), so that all the scheduled jobs
> are run only on the first instance.
>
> I am quite confused by this situation and I really don't know what and
> where to look, to understand what happens.
>
> Does anyone ever experienced a similar issue?

I worked several time with service engine and I interpreted your
feedback as follow :
* the service in the Job List screen, has the state of "Finished" -> Ok
for OFBiz the service return a success
* the flag on the OrderHeader entity has been set to "Y" (by default is
null this field); -> If your service run under transaction, no reason
that your service correctly
* No log file nor zip file is written
   ** I see different case, if you are sure that your code works
properly check if you didn't manage some silent exception ( as
try{...}catch(Exception){ }) and is it's the case review you exception
management.
   ** Increase your code with more log with good log4j configuration for
track your service.
   ** Check if you haven't external element that can be delete your file
by inadvertence
   ** a doubt, send an email with the generate file to your developer
team in // to the storage.
   ** no solution -> Use the remote debugger

I hope you can find a solution to your problem
Nicolas
> Thanks in advance for your help.
>
> Giulio
>

[Giulio Speri - MpStyle Srl]

Hi Nicolas,

thanks for your quick answer.

The service is written in Groovy: this could lead to problems with
exceptions raising?
Do you suggest a Java solution to better manage exceptions?
Since I am one of the developer of the service I can directly work on the
code :) .



2018-03-30 14:18 GMT+02:00 Nicolas Malin <[hidden email]>:

> Hello Giulio,
>
> On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
>
>> Hi all,
>> [...] Now, here is the issue; sometimes (runtime in which happen seems
>> random), I
>> notice that:
>> - the service in the Job List screen, has the state of "Finished" (no
>> errors);
>> - No log file nor zip file is written;
>> - the flag on the OrderHeader entity has been set to "Y" (by default is
>> null this field);
>> - no error or exception found in ofbiz/error log files;
>>
>> The Job Poller on the second instance of ofbiz is disabled
>> ("poll-enable=false" in serviceengine.xml), so that all the scheduled jobs
>> are run only on the first instance.
>>
>> I am quite confused by this situation and I really don't know what and
>> where to look, to understand what happens.
>>
>> Does anyone ever experienced a similar issue?
>>
> I worked several time with service engine and I interpreted your feedback
> as follow :
> * the service in the Job List screen, has the state of "Finished" -> Ok
> for OFBiz the service return a success
> * the flag on the OrderHeader entity has been set to "Y" (by default is
> null this field); -> If your service run under transaction, no reason that
> your service correctly
> * No log file nor zip file is written
>   ** I see different case, if you are sure that your code works properly
> check if you didn't manage some silent exception ( as
> try{...}catch(Exception){ }) and is it's the case review you exception
> management.
>   ** Increase your code with more log with good log4j configuration for
> track your service.
>   ** Check if you haven't external element that can be delete your file by
> inadvertence
>   ** a doubt, send an email with the generate file to your developer team
> in // to the storage.
>   ** no solution -> Use the remote debugger
>
> I hope you can find a solution to your problem
> Nicolas
>
> Thanks in advance for your help.
>>
>> Giulio
>>
>>
>

--
Giulio Speri


*Mp Styl**e Srl*
via Antonio Meucci, 37
41019 Limidi di Soliera (MO)
T 059/684916
M 334/3779851

www.mpstyle.it

[Rishi Solanki]

Giulio,

In addition to what Nicolas suggested please share or look into the
specific code which logs the files and export the orders at ftp locaion.
Probable cause for such error writer create the file at particular location
but could not push the data/file due to some reason but send no
exception/failure.

My guess problem should be related to pushing the exported data and logs to
ftp location and is giving success but not pushing. Kind of case when we
tell the writer object to write and close. Sometime writer.flush() is also
required.

Also if possible at your end then share the relevant code and api you are
using, to better understand the problem.

HTH!


Rishi Solanki
Sr Manager, Enterprise Software Development
HotWax Systems Pvt. Ltd.
Direct: [phone]
http://www.hotwaxsystems.com
www.hotwax.co

On Fri, Mar 30, 2018 at 5:48 PM, Nicolas Malin <[hidden email]>
wrote:

> Hello Giulio,
>
> On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
>
>> Hi all,
>> [...] Now, here is the issue; sometimes (runtime in which happen seems
>> random), I
>> notice that:
>> - the service in the Job List screen, has the state of "Finished" (no
>> errors);
>> - No log file nor zip file is written;
>> - the flag on the OrderHeader entity has been set to "Y" (by default is
>> null this field);
>> - no error or exception found in ofbiz/error log files;
>>
>> The Job Poller on the second instance of ofbiz is disabled
>> ("poll-enable=false" in serviceengine.xml), so that all the scheduled jobs
>> are run only on the first instance.
>>
>> I am quite confused by this situation and I really don't know what and
>> where to look, to understand what happens.
>>
>> Does anyone ever experienced a similar issue?
>>
> I worked several time with service engine and I interpreted your feedback
> as follow :
> * the service in the Job List screen, has the state of "Finished" -> Ok
> for OFBiz the service return a success
> * the flag on the OrderHeader entity has been set to "Y" (by default is
> null this field); -> If your service run under transaction, no reason that
> your service correctly
> * No log file nor zip file is written
>   ** I see different case, if you are sure that your code works properly
> check if you didn't manage some silent exception ( as
> try{...}catch(Exception){ }) and is it's the case review you exception
> management.
>   ** Increase your code with more log with good log4j configuration for
> track your service.
>   ** Check if you haven't external element that can be delete your file by
> inadvertence
>   ** a doubt, send an email with the generate file to your developer team
> in // to the storage.
>   ** no solution -> Use the remote debugger
>
> I hope you can find a solution to your problem
> Nicolas
>
> Thanks in advance for your help.
>>
>> Giulio
>>
>>
>

[Giulio Speri - MpStyle Srl]

Hi Rishi,

thanks for the reply!

The api used for the ftp transfer are the apache FTP api, group
"commons-net" (version 3.5);
The code used to connect and store file via fftp is the following:

ftp = new FTPClient()
ftp.connect "xxx.yyyyyy.ww"
ftp.enterLocalPassiveMode()
ftp.login "username", "password"
ftp.changeWorkingDirectory "INPUT_DIRECTORY"
//put
InputStream file_put= new FileInputStream(directoryZip);
ftp.setFileType(FTP.BINARY_FILE_TYPE);
ftp.storeFile(zipName, file_put);
file_put.close()
ftp.logout()
ftp.disconnect()

Shouldn't Groovy automatically raise exception in case of errors?




2018-03-30 15:07 GMT+02:00 Rishi Solanki <[hidden email]>:

> Giulio,
>
> In addition to what Nicolas suggested please share or look into the
> specific code which logs the files and export the orders at ftp locaion.
> Probable cause for such error writer create the file at particular location
> but could not push the data/file due to some reason but send no
> exception/failure.
>
> My guess problem should be related to pushing the exported data and logs to
> ftp location and is giving success but not pushing. Kind of case when we
> tell the writer object to write and close. Sometime writer.flush() is also
> required.
>
> Also if possible at your end then share the relevant code and api you are
> using, to better understand the problem.
>
> HTH!
>
>
> Rishi Solanki
> Sr Manager, Enterprise Software Development
> HotWax Systems Pvt. Ltd.
> Direct: [phone]
> http://www.hotwaxsystems.com
> www.hotwax.co
>
> On Fri, Mar 30, 2018 at 5:48 PM, Nicolas Malin <[hidden email]>
> wrote:
>
> > Hello Giulio,
> >
> > On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
> >
> >> Hi all,
> >> [...] Now, here is the issue; sometimes (runtime in which happen seems
> >> random), I
> >> notice that:
> >> - the service in the Job List screen, has the state of "Finished" (no
> >> errors);
> >> - No log file nor zip file is written;
> >> - the flag on the OrderHeader entity has been set to "Y" (by default is
> >> null this field);
> >> - no error or exception found in ofbiz/error log files;
> >>
> >> The Job Poller on the second instance of ofbiz is disabled
> >> ("poll-enable=false" in serviceengine.xml), so that all the scheduled
> jobs
> >> are run only on the first instance.
> >>
> >> I am quite confused by this situation and I really don't know what and
> >> where to look, to understand what happens.
> >>
> >> Does anyone ever experienced a similar issue?
> >>
> > I worked several time with service engine and I interpreted your feedback
> > as follow :
> > * the service in the Job List screen, has the state of "Finished" -> Ok
> > for OFBiz the service return a success
> > * the flag on the OrderHeader entity has been set to "Y" (by default is
> > null this field); -> If your service run under transaction, no reason
> that
> > your service correctly
> > * No log file nor zip file is written
> >   ** I see different case, if you are sure that your code works properly
> > check if you didn't manage some silent exception ( as
> > try{...}catch(Exception){ }) and is it's the case review you exception
> > management.
> >   ** Increase your code with more log with good log4j configuration for
> > track your service.
> >   ** Check if you haven't external element that can be delete your file
> by
> > inadvertence
> >   ** a doubt, send an email with the generate file to your developer team
> > in // to the storage.
> >   ** no solution -> Use the remote debugger
> >
> > I hope you can find a solution to your problem
> > Nicolas
> >
> > Thanks in advance for your help.
> >>
> >> Giulio
> >>
> >>
> >
>

--
Giulio Speri


*Mp Styl**e Srl*
via Antonio Meucci, 37
41019 Limidi di Soliera (MO)
T 059/684916
M 334/3779851

www.mpstyle.it

[Michael Brohl-3]

Hi Giulio,

It seems you directly store the generated file to the ftp server, correct?

In such cases, we store the file to be transferred locally in an archive
directory and then transfer it to the ftp server.

You'll see if the file was written and you have a backup in case there
was something wrong with the ftp transfer.

Best regards,

Michael


Am 30.03.18 um 16:16 schrieb Giulio Speri - MpStyle Srl:

> Hi Rishi,
>
> thanks for the reply!
>
> The api used for the ftp transfer are the apache FTP api, group
> "commons-net" (version 3.5);
> The code used to connect and store file via fftp is the following:
>
> ftp = new FTPClient()
> ftp.connect "xxx.yyyyyy.ww"
> ftp.enterLocalPassiveMode()
> ftp.login "username", "password"
> ftp.changeWorkingDirectory "INPUT_DIRECTORY"
> //put
> InputStream file_put= new FileInputStream(directoryZip);
> ftp.setFileType(FTP.BINARY_FILE_TYPE);
> ftp.storeFile(zipName, file_put);
> file_put.close()
> ftp.logout()
> ftp.disconnect()
>
> Shouldn't Groovy automatically raise exception in case of errors?
>
>
>
>
> 2018-03-30 15:07 GMT+02:00 Rishi Solanki <[hidden email]>:
>
>> Giulio,
>>
>> In addition to what Nicolas suggested please share or look into the
>> specific code which logs the files and export the orders at ftp locaion.
>> Probable cause for such error writer create the file at particular location
>> but could not push the data/file due to some reason but send no
>> exception/failure.
>>
>> My guess problem should be related to pushing the exported data and logs to
>> ftp location and is giving success but not pushing. Kind of case when we
>> tell the writer object to write and close. Sometime writer.flush() is also
>> required.
>>
>> Also if possible at your end then share the relevant code and api you are
>> using, to better understand the problem.
>>
>> HTH!
>>
>>
>> Rishi Solanki
>> Sr Manager, Enterprise Software Development
>> HotWax Systems Pvt. Ltd.
>> Direct: [phone]
>> http://www.hotwaxsystems.com
>> www.hotwax.co
>>
>> On Fri, Mar 30, 2018 at 5:48 PM, Nicolas Malin <[hidden email]>
>> wrote:
>>
>>> Hello Giulio,
>>>
>>> On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
>>>
>>>> Hi all,
>>>> [...] Now, here is the issue; sometimes (runtime in which happen seems
>>>> random), I
>>>> notice that:
>>>> - the service in the Job List screen, has the state of "Finished" (no
>>>> errors);
>>>> - No log file nor zip file is written;
>>>> - the flag on the OrderHeader entity has been set to "Y" (by default is
>>>> null this field);
>>>> - no error or exception found in ofbiz/error log files;
>>>>
>>>> The Job Poller on the second instance of ofbiz is disabled
>>>> ("poll-enable=false" in serviceengine.xml), so that all the scheduled
>> jobs
>>>> are run only on the first instance.
>>>>
>>>> I am quite confused by this situation and I really don't know what and
>>>> where to look, to understand what happens.
>>>>
>>>> Does anyone ever experienced a similar issue?
>>>>
>>> I worked several time with service engine and I interpreted your feedback
>>> as follow :
>>> * the service in the Job List screen, has the state of "Finished" -> Ok
>>> for OFBiz the service return a success
>>> * the flag on the OrderHeader entity has been set to "Y" (by default is
>>> null this field); -> If your service run under transaction, no reason
>> that
>>> your service correctly
>>> * No log file nor zip file is written
>>>    ** I see different case, if you are sure that your code works properly
>>> check if you didn't manage some silent exception ( as
>>> try{...}catch(Exception){ }) and is it's the case review you exception
>>> management.
>>>    ** Increase your code with more log with good log4j configuration for
>>> track your service.
>>>    ** Check if you haven't external element that can be delete your file
>> by
>>> inadvertence
>>>    ** a doubt, send an email with the generate file to your developer team
>>> in // to the storage.
>>>    ** no solution -> Use the remote debugger
>>>
>>> I hope you can find a solution to your problem
>>> Nicolas
>>>
>>> Thanks in advance for your help.
>>>> Giulio
>>>>
>>>>
>
>

[Rishi Solanki]

Hi Giulio,

+1 for Michael suggestion, you should try to backup the file locally before
transferring.

Another item I could see in your code (thanks for sharing) that, it just
connect and transfer the file to ftp location. I see many methods with
FTPClient class. One is to set the buffer size.
FTPClient.setBufferSize(int), which set the internal buffer size for
buffered data streams.

As you mentioned initially mostly you got the successful transfer, and
problem occurs very few time. Please consider the size of the file to
transfer and increase its buffer size should resolve your problem. Also
there are many other methods which allows you to get/set the transfer
status and mode etc should help.

Reference:
https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html

HTH!



Rishi Solanki
Sr Manager, Enterprise Software Development
HotWax Systems Pvt. Ltd.
Direct: [phone]
http://www.hotwaxsystems.com
www.hotwax.co

On Fri, Mar 30, 2018 at 10:42 PM, Michael Brohl <[hidden email]>
wrote:

> Hi Giulio,
>
> It seems you directly store the generated file to the ftp server, correct?
>
> In such cases, we store the file to be transferred locally in an archive
> directory and then transfer it to the ftp server.
>
> You'll see if the file was written and you have a backup in case there was
> something wrong with the ftp transfer.
>
> Best regards,
>
> Michael
>
>
> Am 30.03.18 um 16:16 schrieb Giulio Speri - MpStyle Srl:
>
> Hi Rishi,
>>
>> thanks for the reply!
>>
>> The api used for the ftp transfer are the apache FTP api, group
>> "commons-net" (version 3.5);
>> The code used to connect and store file via fftp is the following:
>>
>> ftp = new FTPClient()
>> ftp.connect "xxx.yyyyyy.ww"
>> ftp.enterLocalPassiveMode()
>> ftp.login "username", "password"
>> ftp.changeWorkingDirectory "INPUT_DIRECTORY"
>> //put
>> InputStream file_put= new FileInputStream(directoryZip);
>> ftp.setFileType(FTP.BINARY_FILE_TYPE);
>> ftp.storeFile(zipName, file_put);
>> file_put.close()
>> ftp.logout()
>> ftp.disconnect()
>>
>> Shouldn't Groovy automatically raise exception in case of errors?
>>
>>
>>
>>
>> 2018-03-30 15:07 GMT+02:00 Rishi Solanki <[hidden email]>:
>>
>> Giulio,
>>>
>>> In addition to what Nicolas suggested please share or look into the
>>> specific code which logs the files and export the orders at ftp locaion.
>>> Probable cause for such error writer create the file at particular
>>> location
>>> but could not push the data/file due to some reason but send no
>>> exception/failure.
>>>
>>> My guess problem should be related to pushing the exported data and logs
>>> to
>>> ftp location and is giving success but not pushing. Kind of case when we
>>> tell the writer object to write and close. Sometime writer.flush() is
>>> also
>>> required.
>>>
>>> Also if possible at your end then share the relevant code and api you are
>>> using, to better understand the problem.
>>>
>>> HTH!
>>>
>>>
>>> Rishi Solanki
>>> Sr Manager, Enterprise Software Development
>>> HotWax Systems Pvt. Ltd.
>>> Direct: [phone]
>>> http://www.hotwaxsystems.com
>>> www.hotwax.co
>>>
>>> On Fri, Mar 30, 2018 at 5:48 PM, Nicolas Malin <[hidden email]
>>> >
>>> wrote:
>>>
>>> Hello Giulio,
>>>>
>>>> On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
>>>>
>>>> Hi all,
>>>>> [...] Now, here is the issue; sometimes (runtime in which happen seems
>>>>> random), I
>>>>> notice that:
>>>>> - the service in the Job List screen, has the state of "Finished" (no
>>>>> errors);
>>>>> - No log file nor zip file is written;
>>>>> - the flag on the OrderHeader entity has been set to "Y" (by default is
>>>>> null this field);
>>>>> - no error or exception found in ofbiz/error log files;
>>>>>
>>>>> The Job Poller on the second instance of ofbiz is disabled
>>>>> ("poll-enable=false" in serviceengine.xml), so that all the scheduled
>>>>>
>>>> jobs
>>>
>>>> are run only on the first instance.
>>>>>
>>>>> I am quite confused by this situation and I really don't know what and
>>>>> where to look, to understand what happens.
>>>>>
>>>>> Does anyone ever experienced a similar issue?
>>>>>
>>>>> I worked several time with service engine and I interpreted your
>>>> feedback
>>>> as follow :
>>>> * the service in the Job List screen, has the state of "Finished" -> Ok
>>>> for OFBiz the service return a success
>>>> * the flag on the OrderHeader entity has been set to "Y" (by default is
>>>> null this field); -> If your service run under transaction, no reason
>>>>
>>> that
>>>
>>>> your service correctly
>>>> * No log file nor zip file is written
>>>>    ** I see different case, if you are sure that your code works
>>>> properly
>>>> check if you didn't manage some silent exception ( as
>>>> try{...}catch(Exception){ }) and is it's the case review you exception
>>>> management.
>>>>    ** Increase your code with more log with good log4j configuration for
>>>> track your service.
>>>>    ** Check if you haven't external element that can be delete your file
>>>>
>>> by
>>>
>>>> inadvertence
>>>>    ** a doubt, send an email with the generate file to your developer
>>>> team
>>>> in // to the storage.
>>>>    ** no solution -> Use the remote debugger
>>>>
>>>> I hope you can find a solution to your problem
>>>> Nicolas
>>>>
>>>> Thanks in advance for your help.
>>>>
>>>>> Giulio
>>>>>
>>>>>
>>>>>
>>
>>
>
>

[Nicolas Malin-2]

Hi Giulio,

When we realized ftp transfer, we process as follow :
1. create a content linked to the OFBiz element. In you case we would be
created an OrderContent-Content with all OrderHeader present in the file.
2. Initialize a communicationEvent with the content, a type FTP_TRANSFER
and a contactMechIdTo with the destination ftp location
3. run a batch that transfer all ftp communication
With this process we haven't lost data and when a problem come it's
easily tracking.

Gil Portenseigne create an issue [1] with the code base improvement with
own customer site feedback

Nicolas

[1] https://issues.apache.org/jira/browse/OFBIZ-10245 File transfer
management with communicationEvent and new contactMech FTP_ADDRESS

PS: I send it to dev because my response cover an improvement but it's
also interesting some user ...

On 02/04/2018 09:54, Rishi Solanki wrote:

> Hi Giulio,
>
> +1 for Michael suggestion, you should try to backup the file locally before
> transferring.
>
> Another item I could see in your code (thanks for sharing) that, it just
> connect and transfer the file to ftp location. I see many methods with
> FTPClient class. One is to set the buffer size.
> FTPClient.setBufferSize(int), which set the internal buffer size for
> buffered data streams.
>
> As you mentioned initially mostly you got the successful transfer, and
> problem occurs very few time. Please consider the size of the file to
> transfer and increase its buffer size should resolve your problem. Also
> there are many other methods which allows you to get/set the transfer
> status and mode etc should help.
>
> Reference:
> https://commons.apache.org/proper/commons-net/apidocs/org/apache/commons/net/ftp/FTPClient.html
>
> HTH!
>
>
>
> Rishi Solanki
> Sr Manager, Enterprise Software Development
> HotWax Systems Pvt. Ltd.
> Direct: [phone]
> http://www.hotwaxsystems.com
> www.hotwax.co
>
> On Fri, Mar 30, 2018 at 10:42 PM, Michael Brohl <[hidden email]>
> wrote:
>
>> Hi Giulio,
>>
>> It seems you directly store the generated file to the ftp server, correct?
>>
>> In such cases, we store the file to be transferred locally in an archive
>> directory and then transfer it to the ftp server.
>>
>> You'll see if the file was written and you have a backup in case there was
>> something wrong with the ftp transfer.
>>
>> Best regards,
>>
>> Michael
>>
>>
>> Am 30.03.18 um 16:16 schrieb Giulio Speri - MpStyle Srl:
>>
>> Hi Rishi,
>>> thanks for the reply!
>>>
>>> The api used for the ftp transfer are the apache FTP api, group
>>> "commons-net" (version 3.5);
>>> The code used to connect and store file via fftp is the following:
>>>
>>> ftp = new FTPClient()
>>> ftp.connect "xxx.yyyyyy.ww"
>>> ftp.enterLocalPassiveMode()
>>> ftp.login "username", "password"
>>> ftp.changeWorkingDirectory "INPUT_DIRECTORY"
>>> //put
>>> InputStream file_put= new FileInputStream(directoryZip);
>>> ftp.setFileType(FTP.BINARY_FILE_TYPE);
>>> ftp.storeFile(zipName, file_put);
>>> file_put.close()
>>> ftp.logout()
>>> ftp.disconnect()
>>>
>>> Shouldn't Groovy automatically raise exception in case of errors?
>>>
>>>
>>>
>>>
>>> 2018-03-30 15:07 GMT+02:00 Rishi Solanki <[hidden email]>:
>>>
>>> Giulio,
>>>> In addition to what Nicolas suggested please share or look into the
>>>> specific code which logs the files and export the orders at ftp locaion.
>>>> Probable cause for such error writer create the file at particular
>>>> location
>>>> but could not push the data/file due to some reason but send no
>>>> exception/failure.
>>>>
>>>> My guess problem should be related to pushing the exported data and logs
>>>> to
>>>> ftp location and is giving success but not pushing. Kind of case when we
>>>> tell the writer object to write and close. Sometime writer.flush() is
>>>> also
>>>> required.
>>>>
>>>> Also if possible at your end then share the relevant code and api you are
>>>> using, to better understand the problem.
>>>>
>>>> HTH!
>>>>
>>>>
>>>> Rishi Solanki
>>>> Sr Manager, Enterprise Software Development
>>>> HotWax Systems Pvt. Ltd.
>>>> Direct: [phone]
>>>> http://www.hotwaxsystems.com
>>>> www.hotwax.co
>>>>
>>>> On Fri, Mar 30, 2018 at 5:48 PM, Nicolas Malin <[hidden email]
>>>> wrote:
>>>>
>>>> Hello Giulio,
>>>>> On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
>>>>>
>>>>> Hi all,
>>>>>> [...] Now, here is the issue; sometimes (runtime in which happen seems
>>>>>> random), I
>>>>>> notice that:
>>>>>> - the service in the Job List screen, has the state of "Finished" (no
>>>>>> errors);
>>>>>> - No log file nor zip file is written;
>>>>>> - the flag on the OrderHeader entity has been set to "Y" (by default is
>>>>>> null this field);
>>>>>> - no error or exception found in ofbiz/error log files;
>>>>>>
>>>>>> The Job Poller on the second instance of ofbiz is disabled
>>>>>> ("poll-enable=false" in serviceengine.xml), so that all the scheduled
>>>>>>
>>>>> jobs
>>>>> are run only on the first instance.
>>>>>> I am quite confused by this situation and I really don't know what and
>>>>>> where to look, to understand what happens.
>>>>>>
>>>>>> Does anyone ever experienced a similar issue?
>>>>>>
>>>>>> I worked several time with service engine and I interpreted your
>>>>> feedback
>>>>> as follow :
>>>>> * the service in the Job List screen, has the state of "Finished" -> Ok
>>>>> for OFBiz the service return a success
>>>>> * the flag on the OrderHeader entity has been set to "Y" (by default is
>>>>> null this field); -> If your service run under transaction, no reason
>>>>>
>>>> that
>>>>
>>>>> your service correctly
>>>>> * No log file nor zip file is written
>>>>>     ** I see different case, if you are sure that your code works
>>>>> properly
>>>>> check if you didn't manage some silent exception ( as
>>>>> try{...}catch(Exception){ }) and is it's the case review you exception
>>>>> management.
>>>>>     ** Increase your code with more log with good log4j configuration for
>>>>> track your service.
>>>>>     ** Check if you haven't external element that can be delete your file
>>>>>
>>>> by
>>>>
>>>>> inadvertence
>>>>>     ** a doubt, send an email with the generate file to your developer
>>>>> team
>>>>> in // to the storage.
>>>>>     ** no solution -> Use the remote debugger
>>>>>
>>>>> I hope you can find a solution to your problem
>>>>> Nicolas
>>>>>
>>>>> Thanks in advance for your help.
>>>>>
>>>>>> Giulio
>>>>>>
>>>>>>
>>>>>>
>>>
>>

[Giulio Speri - MpStyle Srl]

Hi Nicolas,

really interesting your solution to have ftp data persisted into the db; it
improves tracking and debugging of the transfers, for sure.
As first step I think that I'll try to tidy up and improve the code of the
service (specially the ftp part), with your previous suggestions and see if
the problem comes to a solution.
If not, I will head for the "db persistency solution".
I'll keep you informed on this.

Thank you all for your help!

Have a good day,
Giulio

2018-04-03 9:28 GMT+02:00 Nicolas Malin <[hidden email]>:

> Hi Giulio,
>
> When we realized ftp transfer, we process as follow :
> 1. create a content linked to the OFBiz element. In you case we would be
> created an OrderContent-Content with all OrderHeader present in the file.
> 2. Initialize a communicationEvent with the content, a type FTP_TRANSFER
> and a contactMechIdTo with the destination ftp location
> 3. run a batch that transfer all ftp communication
> With this process we haven't lost data and when a problem come it's easily
> tracking.
>
> Gil Portenseigne create an issue [1] with the code base improvement with
> own customer site feedback
>
> Nicolas
>
> [1] https://issues.apache.org/jira/browse/OFBIZ-10245 File transfer
> management with communicationEvent and new contactMech FTP_ADDRESS
>
> PS: I send it to dev because my response cover an improvement but it's
> also interesting some user ...
>
> On 02/04/2018 09:54, Rishi Solanki wrote:
>
>> Hi Giulio,
>>
>> +1 for Michael suggestion, you should try to backup the file locally
>> before
>> transferring.
>>
>> Another item I could see in your code (thanks for sharing) that, it just
>> connect and transfer the file to ftp location. I see many methods with
>> FTPClient class. One is to set the buffer size.
>> FTPClient.setBufferSize(int), which set the internal buffer size for
>> buffered data streams.
>>
>> As you mentioned initially mostly you got the successful transfer, and
>> problem occurs very few time. Please consider the size of the file to
>> transfer and increase its buffer size should resolve your problem. Also
>> there are many other methods which allows you to get/set the transfer
>> status and mode etc should help.
>>
>> Reference:
>> https://commons.apache.org/proper/commons-net/apidocs/org/
>> apache/commons/net/ftp/FTPClient.html
>>
>> HTH!
>>
>>
>>
>> Rishi Solanki
>> Sr Manager, Enterprise Software Development
>> HotWax Systems Pvt. Ltd.
>> Direct: [phone]
>> http://www.hotwaxsystems.com
>> www.hotwax.co
>>
>> On Fri, Mar 30, 2018 at 10:42 PM, Michael Brohl <[hidden email]
>> >
>> wrote:
>>
>> Hi Giulio,
>>>
>>> It seems you directly store the generated file to the ftp server,
>>> correct?
>>>
>>> In such cases, we store the file to be transferred locally in an archive
>>> directory and then transfer it to the ftp server.
>>>
>>> You'll see if the file was written and you have a backup in case there
>>> was
>>> something wrong with the ftp transfer.
>>>
>>> Best regards,
>>>
>>> Michael
>>>
>>>
>>> Am 30.03.18 um 16:16 schrieb Giulio Speri - MpStyle Srl:
>>>
>>> Hi Rishi,
>>>
>>>> thanks for the reply!
>>>>
>>>> The api used for the ftp transfer are the apache FTP api, group
>>>> "commons-net" (version 3.5);
>>>> The code used to connect and store file via fftp is the following:
>>>>
>>>> ftp = new FTPClient()
>>>> ftp.connect "xxx.yyyyyy.ww"
>>>> ftp.enterLocalPassiveMode()
>>>> ftp.login "username", "password"
>>>> ftp.changeWorkingDirectory "INPUT_DIRECTORY"
>>>> //put
>>>> InputStream file_put= new FileInputStream(directoryZip);
>>>> ftp.setFileType(FTP.BINARY_FILE_TYPE);
>>>> ftp.storeFile(zipName, file_put);
>>>> file_put.close()
>>>> ftp.logout()
>>>> ftp.disconnect()
>>>>
>>>> Shouldn't Groovy automatically raise exception in case of errors?
>>>>
>>>>
>>>>
>>>>
>>>> 2018-03-30 15:07 GMT+02:00 Rishi Solanki <[hidden email]>:
>>>>
>>>> Giulio,
>>>>
>>>>> In addition to what Nicolas suggested please share or look into the
>>>>> specific code which logs the files and export the orders at ftp
>>>>> locaion.
>>>>> Probable cause for such error writer create the file at particular
>>>>> location
>>>>> but could not push the data/file due to some reason but send no
>>>>> exception/failure.
>>>>>
>>>>> My guess problem should be related to pushing the exported data and
>>>>> logs
>>>>> to
>>>>> ftp location and is giving success but not pushing. Kind of case when
>>>>> we
>>>>> tell the writer object to write and close. Sometime writer.flush() is
>>>>> also
>>>>> required.
>>>>>
>>>>> Also if possible at your end then share the relevant code and api you
>>>>> are
>>>>> using, to better understand the problem.
>>>>>
>>>>> HTH!
>>>>>
>>>>>
>>>>> Rishi Solanki
>>>>> Sr Manager, Enterprise Software Development
>>>>> HotWax Systems Pvt. Ltd.
>>>>> Direct: [phone]
>>>>> http://www.hotwaxsystems.com
>>>>> www.hotwax.co
>>>>>
>>>>> On Fri, Mar 30, 2018 at 5:48 PM, Nicolas Malin <
>>>>> [hidden email]
>>>>> wrote:
>>>>>
>>>>> Hello Giulio,
>>>>>
>>>>>> On 30/03/2018 14:05, Giulio Speri - MpStyle Srl wrote:
>>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>>> [...] Now, here is the issue; sometimes (runtime in which happen
>>>>>>> seems
>>>>>>> random), I
>>>>>>> notice that:
>>>>>>> - the service in the Job List screen, has the state of "Finished" (no
>>>>>>> errors);
>>>>>>> - No log file nor zip file is written;
>>>>>>> - the flag on the OrderHeader entity has been set to "Y" (by default
>>>>>>> is
>>>>>>> null this field);
>>>>>>> - no error or exception found in ofbiz/error log files;
>>>>>>>
>>>>>>> The Job Poller on the second instance of ofbiz is disabled
>>>>>>> ("poll-enable=false" in serviceengine.xml), so that all the scheduled
>>>>>>>
>>>>>>> jobs
>>>>>> are run only on the first instance.
>>>>>>
>>>>>>> I am quite confused by this situation and I really don't know what
>>>>>>> and
>>>>>>> where to look, to understand what happens.
>>>>>>>
>>>>>>> Does anyone ever experienced a similar issue?
>>>>>>>
>>>>>>> I worked several time with service engine and I interpreted your
>>>>>>>
>>>>>> feedback
>>>>>> as follow :
>>>>>> * the service in the Job List screen, has the state of "Finished" ->
>>>>>> Ok
>>>>>> for OFBiz the service return a success
>>>>>> * the flag on the OrderHeader entity has been set to "Y" (by default
>>>>>> is
>>>>>> null this field); -> If your service run under transaction, no reason
>>>>>>
>>>>>> that
>>>>>
>>>>> your service correctly
>>>>>> * No log file nor zip file is written
>>>>>>     ** I see different case, if you are sure that your code works
>>>>>> properly
>>>>>> check if you didn't manage some silent exception ( as
>>>>>> try{...}catch(Exception){ }) and is it's the case review you exception
>>>>>> management.
>>>>>>     ** Increase your code with more log with good log4j configuration
>>>>>> for
>>>>>> track your service.
>>>>>>     ** Check if you haven't external element that can be delete your
>>>>>> file
>>>>>>
>>>>>> by
>>>>>
>>>>> inadvertence
>>>>>>     ** a doubt, send an email with the generate file to your developer
>>>>>> team
>>>>>> in // to the storage.
>>>>>>     ** no solution -> Use the remote debugger
>>>>>>
>>>>>> I hope you can find a solution to your problem
>>>>>> Nicolas
>>>>>>
>>>>>> Thanks in advance for your help.
>>>>>>
>>>>>> Giulio
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>
>>>
>
>

--
Giulio Speri


*Mp Styl**e Srl*
via Antonio Meucci, 37
41019 Limidi di Soliera (MO)
T 059/684916
M 334/3779851

www.mpstyle.it