Hi,
sendEmailDated is scheduled by default to wake up every 15 minutes. I got into situation, that the service woke up, and previous exceution did not finish, hence the file was collected again, what ended out in infinite email sending (i was sending mail to 5000 customers, and it took more than 15 minutes, and each time it was recognized as event that need to be sent). i think the following mechanisms need to be implemented, in order to avoid the communication events to be picked more than once, and in order to make sure, that mail will not be sent twice. 1) adding another state to communicationEvnet, that symbolize that records the message send is being picked up (as of now, the service is picking up In-Progress and change it to Complete when ending), i suggest to add the status "sending", so when the sendEmailDated wakes up, before it starts sending, its taking ownership for it, and the next instance that wakes up, will not pick the communication event. 2) i think that the suggestion i described in #1, is still not good enough, and there should be record level mechanism to avoid multiple sending of same message (e.g. in case the service crushes, and being reexcuted, need to still not send same email twice). The current mechanism, uses same transaction for all the mail that need to be sent. i sugget the following mechanism instead: making new transaction for every individual mail send. the transaction will call the following: create unique constraint on CommunicationEvent for the combination of parentCommEventId and contactMechIdTo. when sending individual message, - create the individual communication event (for each and every recepient,): if its already there, rollback the transaction (this means that the mail already been sent): this will avoid continue send mail in case it was already sent - send the email if there is a problem with sending the mail, rollback the transaction, and then the individual communication event will not be created in this case, we can ensure that the mail will never be sent twice. Comments will be appreciated. Thanks, Amit |
If you're going to send that many emails in one operation, you should be
queuing emails to a LOCAL email service (i.e. postfix on 127.0.0.1). This is overall a good idea: 1) Fast queuing (ofbiz considers delivery a success). 2) You have the ability to 'adjust' the email headers, senders. 3) Background sending/delivery. Some recipients may have bad or non-existent MX records, slow delivery, etc. Let postfix worry about the physical delivery of the email, not ofbiz. Email setup is sort of a black art, but it's the proper way to do this. Seek expert advice if necessary. On Fri, Feb 22, 2019 at 2:45 AM Amit Cahanovich <[hidden email]> wrote: > Hi, > sendEmailDated is scheduled by default to wake up every 15 minutes. > I got into situation, that the service woke up, and previous exceution did > not finish, hence the file was collected again, what ended out in infinite > email sending (i was sending mail to 5000 customers, and it took more than > 15 minutes, and each time it was recognized as event that need to be sent). > > i think the following mechanisms need to be implemented, in order to avoid > the communication events to be picked more than once, and in order to make > sure, that mail will not be sent twice. > > 1) adding another state to communicationEvnet, that symbolize that records > the message send is being picked up (as of now, the service is picking up > In-Progress and change it to Complete when ending), i suggest to add the > status "sending", so when the sendEmailDated wakes up, before it starts > sending, its taking ownership for it, and the next instance that wakes up, > will not pick the communication event. > > 2) i think that the suggestion i described in #1, is still not good enough, > and there should be record level mechanism to avoid multiple sending of > same message (e.g. in case the service crushes, and being reexcuted, need > to still not send same email twice). > The current mechanism, uses same transaction for all the mail that need to > be sent. > i sugget the following mechanism instead: > > making new transaction for every individual mail send. the transaction will > call the following: > create unique constraint on CommunicationEvent for the combination of > parentCommEventId > and contactMechIdTo. > when sending individual message, > > - create the individual communication event (for each and every > recepient,): > if its already there, rollback the transaction (this means that the mail > already been sent): this will avoid continue send mail in case it was > already sent > - send the email > if there is a problem with sending the mail, rollback the transaction, and > then the individual communication event will not be created > in this case, we can ensure that the mail will never be sent twice. > > Comments will be appreciated. > Thanks, > Amit > |
Thanks for the advise, i suppose its refering point #2, and i will
definitely do so. but what about point#1, adding additional state, in order to avoid next sendEmailDated job instance picking same communication event? בתאריך יום ו׳, 22 בפבר׳ 2019, 17:55, מאת Mike <[hidden email]>: > If you're going to send that many emails in one operation, you should be > queuing emails to a LOCAL email service (i.e. postfix on 127.0.0.1). This > is overall a good idea: > > 1) Fast queuing (ofbiz considers delivery a success). > 2) You have the ability to 'adjust' the email headers, senders. > 3) Background sending/delivery. > > Some recipients may have bad or non-existent MX records, slow delivery, > etc. Let postfix worry about the physical delivery of the email, not > ofbiz. Email setup is sort of a black art, but it's the proper way to do > this. Seek expert advice if necessary. > > > > > > On Fri, Feb 22, 2019 at 2:45 AM Amit Cahanovich <[hidden email]> > wrote: > > > Hi, > > sendEmailDated is scheduled by default to wake up every 15 minutes. > > I got into situation, that the service woke up, and previous exceution > did > > not finish, hence the file was collected again, what ended out in > infinite > > email sending (i was sending mail to 5000 customers, and it took more > than > > 15 minutes, and each time it was recognized as event that need to be > sent). > > > > i think the following mechanisms need to be implemented, in order to > avoid > > the communication events to be picked more than once, and in order to > make > > sure, that mail will not be sent twice. > > > > 1) adding another state to communicationEvnet, that symbolize that > records > > the message send is being picked up (as of now, the service is picking up > > In-Progress and change it to Complete when ending), i suggest to add the > > status "sending", so when the sendEmailDated wakes up, before it starts > > sending, its taking ownership for it, and the next instance that wakes > up, > > will not pick the communication event. > > > > 2) i think that the suggestion i described in #1, is still not good > enough, > > and there should be record level mechanism to avoid multiple sending of > > same message (e.g. in case the service crushes, and being reexcuted, need > > to still not send same email twice). > > The current mechanism, uses same transaction for all the mail that need > to > > be sent. > > i sugget the following mechanism instead: > > > > making new transaction for every individual mail send. the transaction > will > > call the following: > > create unique constraint on CommunicationEvent for the combination of > > parentCommEventId > > and contactMechIdTo. > > when sending individual message, > > > > - create the individual communication event (for each and every > > recepient,): > > if its already there, rollback the transaction (this means that the mail > > already been sent): this will avoid continue send mail in case it was > > already sent > > - send the email > > if there is a problem with sending the mail, rollback the transaction, > and > > then the individual communication event will not be created > > in this case, we can ensure that the mail will never be sent twice. > > > > Comments will be appreciated. > > Thanks, > > Amit > > > |
The email deliveries will be so fast (from an ofbiz perspective) when
queuing locally that it'll be a non-factor. You can queue 10 of thousands of emails per minute. You don't want to waste valuable java threads physically sending email. On Fri, Feb 22, 2019 at 8:30 AM Amit Cahanovich <[hidden email]> wrote: > Thanks for the advise, i suppose its refering point #2, and i will > definitely do so. but what about point#1, adding additional state, in order > to avoid next sendEmailDated job instance picking same communication event? > > בתאריך יום ו׳, 22 בפבר׳ 2019, 17:55, מאת Mike <[hidden email]>: > > > If you're going to send that many emails in one operation, you should be > > queuing emails to a LOCAL email service (i.e. postfix on 127.0.0.1). > This > > is overall a good idea: > > > > 1) Fast queuing (ofbiz considers delivery a success). > > 2) You have the ability to 'adjust' the email headers, senders. > > 3) Background sending/delivery. > > > > Some recipients may have bad or non-existent MX records, slow delivery, > > etc. Let postfix worry about the physical delivery of the email, not > > ofbiz. Email setup is sort of a black art, but it's the proper way to do > > this. Seek expert advice if necessary. > > > > > > > > > > > > On Fri, Feb 22, 2019 at 2:45 AM Amit Cahanovich <[hidden email]> > > wrote: > > > > > Hi, > > > sendEmailDated is scheduled by default to wake up every 15 minutes. > > > I got into situation, that the service woke up, and previous exceution > > did > > > not finish, hence the file was collected again, what ended out in > > infinite > > > email sending (i was sending mail to 5000 customers, and it took more > > than > > > 15 minutes, and each time it was recognized as event that need to be > > sent). > > > > > > i think the following mechanisms need to be implemented, in order to > > avoid > > > the communication events to be picked more than once, and in order to > > make > > > sure, that mail will not be sent twice. > > > > > > 1) adding another state to communicationEvnet, that symbolize that > > records > > > the message send is being picked up (as of now, the service is picking > up > > > In-Progress and change it to Complete when ending), i suggest to add > the > > > status "sending", so when the sendEmailDated wakes up, before it starts > > > sending, its taking ownership for it, and the next instance that wakes > > up, > > > will not pick the communication event. > > > > > > 2) i think that the suggestion i described in #1, is still not good > > enough, > > > and there should be record level mechanism to avoid multiple sending of > > > same message (e.g. in case the service crushes, and being reexcuted, > need > > > to still not send same email twice). > > > The current mechanism, uses same transaction for all the mail that need > > to > > > be sent. > > > i sugget the following mechanism instead: > > > > > > making new transaction for every individual mail send. the transaction > > will > > > call the following: > > > create unique constraint on CommunicationEvent for the combination of > > > parentCommEventId > > > and contactMechIdTo. > > > when sending individual message, > > > > > > - create the individual communication event (for each and every > > > recepient,): > > > if its already there, rollback the transaction (this means that the > > > already been sent): this will avoid continue send mail in case it was > > > already sent > > > - send the email > > > if there is a problem with sending the mail, rollback the transaction, > > and > > > then the individual communication event will not be created > > > in this case, we can ensure that the mail will never be sent twice. > > > > > > Comments will be appreciated. > > > Thanks, > > > Amit > > > > > > |
Actually i do it on a separate batch instance that is dedicated for all
async jobs. The online activity is not affected, and i use the same mechanism also to send sms's, calling 3rd party that i must submit them one by one. so i must use the java threads for it. I thought maybe others would be able to benefit from this kind of deployment, and i see great advantage for having same mechanism regardless the notification channel. In addition, personally, i'm very conservative about an assumption of services will finish on time. Murphy always live on my servers 🌝 בתאריך יום ו׳, 22 בפבר׳ 2019, 18:46, מאת Mike <[hidden email]>: > The email deliveries will be so fast (from an ofbiz perspective) when > queuing locally that it'll be a non-factor. You can queue 10 of thousands > of emails per minute. You don't want to waste valuable java threads > physically sending email. > > On Fri, Feb 22, 2019 at 8:30 AM Amit Cahanovich <[hidden email]> > wrote: > > > Thanks for the advise, i suppose its refering point #2, and i will > > definitely do so. but what about point#1, adding additional state, in > order > > to avoid next sendEmailDated job instance picking same communication > event? > > > > בתאריך יום ו׳, 22 בפבר׳ 2019, 17:55, מאת Mike <[hidden email]>: > > > > > If you're going to send that many emails in one operation, you should > be > > > queuing emails to a LOCAL email service (i.e. postfix on 127.0.0.1). > > This > > > is overall a good idea: > > > > > > 1) Fast queuing (ofbiz considers delivery a success). > > > 2) You have the ability to 'adjust' the email headers, senders. > > > 3) Background sending/delivery. > > > > > > Some recipients may have bad or non-existent MX records, slow delivery, > > > etc. Let postfix worry about the physical delivery of the email, not > > > ofbiz. Email setup is sort of a black art, but it's the proper way to > do > > > this. Seek expert advice if necessary. > > > > > > > > > > > > > > > > > > On Fri, Feb 22, 2019 at 2:45 AM Amit Cahanovich <[hidden email]> > > > wrote: > > > > > > > Hi, > > > > sendEmailDated is scheduled by default to wake up every 15 minutes. > > > > I got into situation, that the service woke up, and previous > exceution > > > did > > > > not finish, hence the file was collected again, what ended out in > > > infinite > > > > email sending (i was sending mail to 5000 customers, and it took more > > > than > > > > 15 minutes, and each time it was recognized as event that need to be > > > sent). > > > > > > > > i think the following mechanisms need to be implemented, in order to > > > avoid > > > > the communication events to be picked more than once, and in order to > > > make > > > > sure, that mail will not be sent twice. > > > > > > > > 1) adding another state to communicationEvnet, that symbolize that > > > records > > > > the message send is being picked up (as of now, the service is > picking > > up > > > > In-Progress and change it to Complete when ending), i suggest to add > > the > > > > status "sending", so when the sendEmailDated wakes up, before it > starts > > > > sending, its taking ownership for it, and the next instance that > wakes > > > up, > > > > will not pick the communication event. > > > > > > > > 2) i think that the suggestion i described in #1, is still not good > > > enough, > > > > and there should be record level mechanism to avoid multiple sending > of > > > > same message (e.g. in case the service crushes, and being reexcuted, > > need > > > > to still not send same email twice). > > > > The current mechanism, uses same transaction for all the mail that > need > > > to > > > > be sent. > > > > i sugget the following mechanism instead: > > > > > > > > making new transaction for every individual mail send. the > transaction > > > will > > > > call the following: > > > > create unique constraint on CommunicationEvent for the combination > of > > > > parentCommEventId > > > > and contactMechIdTo. > > > > when sending individual message, > > > > > > > > - create the individual communication event (for each and every > > > > recepient,): > > > > if its already there, rollback the transaction (this means that the > > > > already been sent): this will avoid continue send mail in case it was > > > > already sent > > > > - send the email > > > > if there is a problem with sending the mail, rollback the > transaction, > > > and > > > > then the individual communication event will not be created > > > > in this case, we can ensure that the mail will never be sent twice. > > > > > > > > Comments will be appreciated. > > > > Thanks, > > > > Amit > > > > > > > > > > |
Hi Amit,
you can configure the service so that it will not be executed if the previous one is still running. See the keyword "semaphore" on the service engine guide in our wiki [1]. Regards, Michael Brohl ecomify GmbH www.ecomify.de [1] https://cwiki.apache.org/confluence/display/OFBIZ/Service+Engine+Guide Am 22.02.19 um 18:24 schrieb Amit Cahanovich: > Actually i do it on a separate batch instance that is dedicated for all > async jobs. The online activity is not affected, and i use the same > mechanism also to send sms's, calling 3rd party that i must submit them one > by one. so i must use the java threads for it. > I thought maybe others would be able to benefit from this kind of > deployment, and i see great advantage for having same mechanism regardless > the notification channel. > In addition, personally, i'm very conservative about an assumption of > services will finish on time. Murphy always live on my servers 🌝 > > בתאריך יום ו׳, 22 בפבר׳ 2019, 18:46, מאת Mike <[hidden email]>: > >> The email deliveries will be so fast (from an ofbiz perspective) when >> queuing locally that it'll be a non-factor. You can queue 10 of thousands >> of emails per minute. You don't want to waste valuable java threads >> physically sending email. >> >> On Fri, Feb 22, 2019 at 8:30 AM Amit Cahanovich <[hidden email]> >> wrote: >> >>> Thanks for the advise, i suppose its refering point #2, and i will >>> definitely do so. but what about point#1, adding additional state, in >> order >>> to avoid next sendEmailDated job instance picking same communication >> event? >>> בתאריך יום ו׳, 22 בפבר׳ 2019, 17:55, מאת Mike <[hidden email]>: >>> >>>> If you're going to send that many emails in one operation, you should >> be >>>> queuing emails to a LOCAL email service (i.e. postfix on 127.0.0.1). >>> This >>>> is overall a good idea: >>>> >>>> 1) Fast queuing (ofbiz considers delivery a success). >>>> 2) You have the ability to 'adjust' the email headers, senders. >>>> 3) Background sending/delivery. >>>> >>>> Some recipients may have bad or non-existent MX records, slow delivery, >>>> etc. Let postfix worry about the physical delivery of the email, not >>>> ofbiz. Email setup is sort of a black art, but it's the proper way to >> do >>>> this. Seek expert advice if necessary. >>>> >>>> >>>> >>>> >>>> >>>> On Fri, Feb 22, 2019 at 2:45 AM Amit Cahanovich <[hidden email]> >>>> wrote: >>>> >>>>> Hi, >>>>> sendEmailDated is scheduled by default to wake up every 15 minutes. >>>>> I got into situation, that the service woke up, and previous >> exceution >>>> did >>>>> not finish, hence the file was collected again, what ended out in >>>> infinite >>>>> email sending (i was sending mail to 5000 customers, and it took more >>>> than >>>>> 15 minutes, and each time it was recognized as event that need to be >>>> sent). >>>>> i think the following mechanisms need to be implemented, in order to >>>> avoid >>>>> the communication events to be picked more than once, and in order to >>>> make >>>>> sure, that mail will not be sent twice. >>>>> >>>>> 1) adding another state to communicationEvnet, that symbolize that >>>> records >>>>> the message send is being picked up (as of now, the service is >> picking >>> up >>>>> In-Progress and change it to Complete when ending), i suggest to add >>> the >>>>> status "sending", so when the sendEmailDated wakes up, before it >> starts >>>>> sending, its taking ownership for it, and the next instance that >> wakes >>>> up, >>>>> will not pick the communication event. >>>>> >>>>> 2) i think that the suggestion i described in #1, is still not good >>>> enough, >>>>> and there should be record level mechanism to avoid multiple sending >> of >>>>> same message (e.g. in case the service crushes, and being reexcuted, >>> need >>>>> to still not send same email twice). >>>>> The current mechanism, uses same transaction for all the mail that >> need >>>> to >>>>> be sent. >>>>> i sugget the following mechanism instead: >>>>> >>>>> making new transaction for every individual mail send. the >> transaction >>>> will >>>>> call the following: >>>>> create unique constraint on CommunicationEvent for the combination >> of >>>>> parentCommEventId >>>>> and contactMechIdTo. >>>>> when sending individual message, >>>>> >>>>> - create the individual communication event (for each and every >>>>> recepient,): >>>>> if its already there, rollback the transaction (this means that the >>>>> already been sent): this will avoid continue send mail in case it was >>>>> already sent >>>>> - send the email >>>>> if there is a problem with sending the mail, rollback the >> transaction, >>>> and >>>>> then the individual communication event will not be created >>>>> in this case, we can ensure that the mail will never be sent twice. >>>>> >>>>> Comments will be appreciated. >>>>> Thanks, >>>>> Amit >>>>> smime.p7s (5K) Download Attachment |
thanks Michael, this what i needed :-)
On Sat, Feb 23, 2019 at 3:14 AM Michael Brohl <[hidden email]> wrote: > Hi Amit, > > you can configure the service so that it will not be executed if the > previous one is still running. > > See the keyword "semaphore" on the service engine guide in our wiki [1]. > > Regards, > > Michael Brohl > ecomify GmbH > www.ecomify.de > > > [1] https://cwiki.apache.org/confluence/display/OFBIZ/Service+Engine+Guide > > > Am 22.02.19 um 18:24 schrieb Amit Cahanovich: > > Actually i do it on a separate batch instance that is dedicated for all > > async jobs. The online activity is not affected, and i use the same > > mechanism also to send sms's, calling 3rd party that i must submit them > one > > by one. so i must use the java threads for it. > > I thought maybe others would be able to benefit from this kind of > > deployment, and i see great advantage for having same mechanism > regardless > > the notification channel. > > In addition, personally, i'm very conservative about an assumption of > > services will finish on time. Murphy always live on my servers 🌝 > > > > בתאריך יום ו׳, 22 בפבר׳ 2019, 18:46, מאת Mike <[hidden email]>: > > > >> The email deliveries will be so fast (from an ofbiz perspective) when > >> queuing locally that it'll be a non-factor. You can queue 10 of > thousands > >> of emails per minute. You don't want to waste valuable java threads > >> physically sending email. > >> > >> On Fri, Feb 22, 2019 at 8:30 AM Amit Cahanovich <[hidden email]> > >> wrote: > >> > >>> Thanks for the advise, i suppose its refering point #2, and i will > >>> definitely do so. but what about point#1, adding additional state, in > >> order > >>> to avoid next sendEmailDated job instance picking same communication > >> event? > >>> בתאריך יום ו׳, 22 בפבר׳ 2019, 17:55, מאת Mike <[hidden email]>: > >>> > >>>> If you're going to send that many emails in one operation, you should > >> be > >>>> queuing emails to a LOCAL email service (i.e. postfix on 127.0.0.1). > >>> This > >>>> is overall a good idea: > >>>> > >>>> 1) Fast queuing (ofbiz considers delivery a success). > >>>> 2) You have the ability to 'adjust' the email headers, senders. > >>>> 3) Background sending/delivery. > >>>> > >>>> Some recipients may have bad or non-existent MX records, slow > delivery, > >>>> etc. Let postfix worry about the physical delivery of the email, not > >>>> ofbiz. Email setup is sort of a black art, but it's the proper way to > >> do > >>>> this. Seek expert advice if necessary. > >>>> > >>>> > >>>> > >>>> > >>>> > >>>> On Fri, Feb 22, 2019 at 2:45 AM Amit Cahanovich <[hidden email]> > >>>> wrote: > >>>> > >>>>> Hi, > >>>>> sendEmailDated is scheduled by default to wake up every 15 minutes. > >>>>> I got into situation, that the service woke up, and previous > >> exceution > >>>> did > >>>>> not finish, hence the file was collected again, what ended out in > >>>> infinite > >>>>> email sending (i was sending mail to 5000 customers, and it took more > >>>> than > >>>>> 15 minutes, and each time it was recognized as event that need to be > >>>> sent). > >>>>> i think the following mechanisms need to be implemented, in order to > >>>> avoid > >>>>> the communication events to be picked more than once, and in order to > >>>> make > >>>>> sure, that mail will not be sent twice. > >>>>> > >>>>> 1) adding another state to communicationEvnet, that symbolize that > >>>> records > >>>>> the message send is being picked up (as of now, the service is > >> picking > >>> up > >>>>> In-Progress and change it to Complete when ending), i suggest to add > >>> the > >>>>> status "sending", so when the sendEmailDated wakes up, before it > >> starts > >>>>> sending, its taking ownership for it, and the next instance that > >> wakes > >>>> up, > >>>>> will not pick the communication event. > >>>>> > >>>>> 2) i think that the suggestion i described in #1, is still not good > >>>> enough, > >>>>> and there should be record level mechanism to avoid multiple sending > >> of > >>>>> same message (e.g. in case the service crushes, and being reexcuted, > >>> need > >>>>> to still not send same email twice). > >>>>> The current mechanism, uses same transaction for all the mail that > >> need > >>>> to > >>>>> be sent. > >>>>> i sugget the following mechanism instead: > >>>>> > >>>>> making new transaction for every individual mail send. the > >> transaction > >>>> will > >>>>> call the following: > >>>>> create unique constraint on CommunicationEvent for the combination > >> of > >>>>> parentCommEventId > >>>>> and contactMechIdTo. > >>>>> when sending individual message, > >>>>> > >>>>> - create the individual communication event (for each and every > >>>>> recepient,): > >>>>> if its already there, rollback the transaction (this means that the > >>>>> already been sent): this will avoid continue send mail in case it was > >>>>> already sent > >>>>> - send the email > >>>>> if there is a problem with sending the mail, rollback the > >> transaction, > >>>> and > >>>>> then the individual communication event will not be created > >>>>> in this case, we can ensure that the mail will never be sent twice. > >>>>> > >>>>> Comments will be appreciated. > >>>>> Thanks, > >>>>> Amit > >>>>> > > |
Free forum by Nabble | Edit this page |