Using plain Groovy classes for services.

[Mathieu Lirzin]

Hello,

While I think using Groovy for implementing services is a better choice
than Java, I am not convinced by the rationale of using Groovy DSL
features. Here are the various drawbacks I see:

  - The service DSL breaks the function/method local scoping goodness by
    introducing various global variables (parameters, delegator, ...)

  - There is no clear disctinction between services and helper methods
    (private/public)

  - When Unit testing a helper method defined in a DSL script, a
    cumbersome mechanism for accessing the groovy method is required
   
  - DSL implicit class context abstraction is leaky, for example it is
    hard to understand what is the proper way to define a variable
    outside of a method and how to refer to it from this method.

    def foo = "foo"
    def barService() {
        print foo // => ERROR: No such property: foo for class: ...
    }

IMO DSL features introduce accidental complexity with very little
benefits.  Since plain Groovy classes doesn't suffer from the downsides
I described above and preserve the Groovy goodness (map literals,
optional typing, ...), I recommend transitioning from DSL scripts to
classes.

What do people think?

Thanks.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Zhang Wei]

+1,

We can also use groovy for other java classes
________________________________
发件人: Mathieu Lirzin <[hidden email]>
发送时间: 2018年11月11日 23:32
收件人: OFBIZ Development Mailing List
主题: Using plain Groovy classes for services.

Hello,

While I think using Groovy for implementing services is a better choice
than Java, I am not convinced by the rationale of using Groovy DSL
features. Here are the various drawbacks I see:

  - The service DSL breaks the function/method local scoping goodness by
    introducing various global variables (parameters, delegator, ...)

  - There is no clear disctinction between services and helper methods
    (private/public)

  - When Unit testing a helper method defined in a DSL script, a
    cumbersome mechanism for accessing the groovy method is required

  - DSL implicit class context abstraction is leaky, for example it is
    hard to understand what is the proper way to define a variable
    outside of a method and how to refer to it from this method.

    def foo = "foo"
    def barService() {
        print foo // => ERROR: No such property: foo for class: ...
    }

IMO DSL features introduce accidental complexity with very little
benefits.  Since plain Groovy classes doesn't suffer from the downsides
I described above and preserve the Groovy goodness (map literals,
optional typing, ...), I recommend transitioning from DSL scripts to
classes.

What do people think?

Thanks.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[taher]

Hi Mathieu,

Can you explain hat you want to do exactly? How do these services
access the delegator and whatnot?
On Sun, Nov 11, 2018 at 5:32 PM Mathieu Lirzin
<[hidden email]> wrote:

>
> Hello,
>
> While I think using Groovy for implementing services is a better choice
> than Java, I am not convinced by the rationale of using Groovy DSL
> features. Here are the various drawbacks I see:
>
>   - The service DSL breaks the function/method local scoping goodness by
>     introducing various global variables (parameters, delegator, ...)
>
>   - There is no clear disctinction between services and helper methods
>     (private/public)
>
>   - When Unit testing a helper method defined in a DSL script, a
>     cumbersome mechanism for accessing the groovy method is required
>
>   - DSL implicit class context abstraction is leaky, for example it is
>     hard to understand what is the proper way to define a variable
>     outside of a method and how to refer to it from this method.
>
>     def foo = "foo"
>     def barService() {
>         print foo // => ERROR: No such property: foo for class: ...
>     }
>
> IMO DSL features introduce accidental complexity with very little
> benefits.  Since plain Groovy classes doesn't suffer from the downsides
> I described above and preserve the Groovy goodness (map literals,
> optional typing, ...), I recommend transitioning from DSL scripts to
> classes.
>
> What do people think?
>
> Thanks.
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Mathieu Lirzin]

Hello Taher,

Taher Alkhateeb <[hidden email]> writes:

> Can you explain hat you want to do exactly? How do these services
> access the delegator and whatnot?

Basically what I have in mind, is to organize services in classes like
what is done in Java with the same method signature:

  (Map ⨯ Map) → Map

where the maps parameters correspond respectively to the “dispatch
context” and the “service input”.  The main difference with Java
services is the possibility to write map and list literals, and avoid
explicit typing with ‘def’.  Additionally it would be easy to provide a
method for the dispatcher which could reads like the current ‘run’
method of the DSL and provide the same semantics (throwing an exception
upon failure).  Something like:

  dispatcher.run service: "fooService" with: [...]

Thanks for your questions.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Jacopo Cappellato-5]

Hi Mathieu,

thank you for starting this interesting conversation.
I think it is fine to implement services in plain Java or in plain Groovy
methods and you have highlighted some of the advantages over their
implementation using the Groovy DSL.
However in my opinion the Groovy DSL (even in its current "basic" version,
implemented thru a few lines of code, that could be enhanced and extended)
has some advantages too and may be preferred by a different audience of
"users" that are more focused on business rules than on programming; data
preparation scripts are also a good fit for the DSL.

Just my two cents,

Jacopo

On Sun, Nov 11, 2018 at 8:02 PM Mathieu Lirzin <[hidden email]>
wrote:

> Hello,
>
> While I think using Groovy for implementing services is a better choice
> than Java, I am not convinced by the rationale of using Groovy DSL
> features. Here are the various drawbacks I see:
>
>   - The service DSL breaks the function/method local scoping goodness by
>     introducing various global variables (parameters, delegator, ...)
>
>   - There is no clear disctinction between services and helper methods
>     (private/public)
>
>   - When Unit testing a helper method defined in a DSL script, a
>     cumbersome mechanism for accessing the groovy method is required
>
>   - DSL implicit class context abstraction is leaky, for example it is
>     hard to understand what is the proper way to define a variable
>     outside of a method and how to refer to it from this method.
>
>     def foo = "foo"
>     def barService() {
>         print foo // => ERROR: No such property: foo for class: ...
>     }
>
> IMO DSL features introduce accidental complexity with very little
> benefits.  Since plain Groovy classes doesn't suffer from the downsides
> I described above and preserve the Groovy goodness (map literals,
> optional typing, ...), I recommend transitioning from DSL scripts to
> classes.
>
> What do people think?
>
> Thanks.
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37
>

[Mathieu Lirzin]

Hello Jacopo,

Jacopo Cappellato <[hidden email]> writes:

> thank you for starting this interesting conversation.
> I think it is fine to implement services in plain Java or in plain Groovy
> methods and you have highlighted some of the advantages over their
> implementation using the Groovy DSL.
> However in my opinion the Groovy DSL (even in its current "basic" version,
> implemented thru a few lines of code, that could be enhanced and extended)
> has some advantages too and may be preferred by a different audience of
> "users" that are more focused on business rules than on programming; data
> preparation scripts are also a good fit for the DSL.

Sure, allowing “business oriented” people to adapt OFBiz to their needs
by letting them to automate a process in terms of business rules is
*very* valuable.

I never had the chance to exchange with people focused on business rules
working with OFBiz which are able to write services/ECA/handlers.
However I am rather skeptic regarding your claim that the Groovy service
DSL allows a wider audience to adapt/compose OFBiz services to their
needs.  I guess it serves more as a “fun” thing for programmers to play
with, than something with an effective business value.

In my opinion improving simplicity (locality, uniformity, value
orientation, ...)  would be far more effective at empowering both
business and code oriented programmers. [1]

Thanks for sharing your view.

[1] https://www.infoq.com/presentations/Simple-Made-Easy-QCon-London-2012

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[taher]

So I think we cannot look at things purely from a theoretical point of
view. There has to be a balance between clean code and continued
usability.

Pros of your approach:
- cleaner semantics
- less pollution of the global namespace
- the other benefits you mentioned

Cons of your approach:
- less orientation towards business focused individuals
- more work on part of the developer. The DSL was always historically
a comfort point for OFBiz not just the groovy DSL but XML and
everything else.
- Major amount of work to refactor all the services.

I'm also not sure the problems you're facing are really that critical.
For example, you mentioned a con in unit-testing support functions?
Really? It's a support function for a service, you _must_ have an
integration test for that thing anyway. You cannot apply TDD on
services, the nature of it is just not workable. And If you add unit
tests your support functions you have now mixed production with test
code without an obvious advantage (TDD is about millisecond tests).

So weighing the pros and cons, and listening also to Jacopo's
feedback, I'm not sure this would be a highly valuable move.
On Wed, Nov 14, 2018 at 7:35 PM Mathieu Lirzin
<[hidden email]> wrote:

>
> Hello Jacopo,
>
> Jacopo Cappellato <[hidden email]> writes:
>
> > thank you for starting this interesting conversation.
> > I think it is fine to implement services in plain Java or in plain Groovy
> > methods and you have highlighted some of the advantages over their
> > implementation using the Groovy DSL.
> > However in my opinion the Groovy DSL (even in its current "basic" version,
> > implemented thru a few lines of code, that could be enhanced and extended)
> > has some advantages too and may be preferred by a different audience of
> > "users" that are more focused on business rules than on programming; data
> > preparation scripts are also a good fit for the DSL.
>
> Sure, allowing “business oriented” people to adapt OFBiz to their needs
> by letting them to automate a process in terms of business rules is
> *very* valuable.
>
> I never had the chance to exchange with people focused on business rules
> working with OFBiz which are able to write services/ECA/handlers.
> However I am rather skeptic regarding your claim that the Groovy service
> DSL allows a wider audience to adapt/compose OFBiz services to their
> needs.  I guess it serves more as a “fun” thing for programmers to play
> with, than something with an effective business value.
>
> In my opinion improving simplicity (locality, uniformity, value
> orientation, ...)  would be far more effective at empowering both
> business and code oriented programmers. [1]
>
> Thanks for sharing your view.
>
> [1] https://www.infoq.com/presentations/Simple-Made-Easy-QCon-London-2012
>
> --
> Mathieu Lirzin
> GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Mathieu Lirzin]

Hello Taher,

Taher Alkhateeb <[hidden email]> writes:

> So I think we cannot look at things purely from a theoretical point of
> view. There has to be a balance between clean code and continued
> usability.
>
> Pros of your approach:
> - cleaner semantics
> - less pollution of the global namespace
> - the other benefits you mentioned
>
> Cons of your approach:
> - less orientation towards business focused individuals

The link between "business orientation" and the use of Groovy DSL is
only theoric.

> - more work on part of the developer. The DSL was always historically
> a comfort point for OFBiz not just the groovy DSL but XML and
> everything else.
> - Major amount of work to refactor all the services.

The amount of work is minimal in comparison to the effort of migrating
from Minilang to Groovy, since this is basically a matter of changing
the signature of the Groovy service and extracting the delegator and
dispatcher for the dispatch context parameter.

> I'm also not sure the problems you're facing are really that critical.

To make it clear this is not *my* problem.  As a professional programmer
I can live with this extra incidental complexity which only slows me
down.  The problem is in fact the one of the business people which have
to pay for this extra time.  :-)

> For example, you mentioned a con in unit-testing support functions?
> Really? It's a support function for a service, you _must_ have an
> integration test for that thing anyway. You cannot apply TDD on
> services, the nature of it is just not workable. And If you add unit
> tests your support functions you have now mixed production with test
> code without an obvious advantage (TDD is about millisecond tests).

The con I mentioned is theoric since, my experience in regard of unit
testing Groovy DSL code is based on testing the ‘GetLocaleList’ action
for the ‘LookupLocale’ screen.

For services I think the business logic and validation of data contained
in services could often be extracted in separate unit testable methods
which would allow cheap better case coverage to complement the
integration test which is about checking that the service has the
expected effect on the database.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37

[Jacques Le Roux]

Le 17/11/2018 à 20:12, Mathieu Lirzin a écrit :
> For services I think the business logic and validation of data contained
> in services could often be extracted in separate unit testable methods
> which would allow cheap better case coverage to complement the
> integration test which is about checking that the service has the
> expected effect on the database.
Hi Mathieu,

I see only one possible tiny drawback with this solution.

I don't know you but most of the time, for small changes, I don't launch the integration tests, they are too long.
I wait for Buildbot feedback, which is most of the time not surprisingly good.

Now every time we build OFBiz, the unit tests pass. Currently we have around thirty of them and they pass very quickly.
How many time it will take when we will have hundreds, if not thousands of them?

Even if it's a small population, maybe we can already estimate an average time from the thirty we have?

Thanks

Jacques

[Jacques Le Roux]

Le 21/11/2018 à 17:43, Jacques Le Roux a écrit :

> Le 17/11/2018 à 20:12, Mathieu Lirzin a écrit :
>> For services I think the business logic and validation of data contained
>> in services could often be extracted in separate unit testable methods
>> which would allow cheap better case coverage to complement the
>> integration test which is about checking that the service has the
>> expected effect on the database.
> Hi Mathieu,
>
> I see only one possible tiny drawback with this solution.
>
> I don't know you but most of the time, for small changes, I don't launch the integration tests, they are too long.
> I wait for Buildbot feedback, which is most of the time not surprisingly good.
>
> Now every time we build OFBiz, the unit tests pass. Currently we have around thirty of them and they pass very quickly.
> How many time it will take when we will have hundreds, if not thousands of them?
>
> Even if it's a small population, maybe we can already estimate an average time from the thirty we have?
>
> Thanks
>
> Jacques
>
>

OK, I can answer to myself on this. This is not a problem at all.
If someone thinks the test are too long in a particular circumstance s/he can use the same strategy that I often use for integration tests: skip them
with "build -x test" and let Buildbot do the the work.

Jacques

[Mathieu Lirzin]

Hello Jacques,

Jacques Le Roux <[hidden email]> writes:

> Le 21/11/2018 à 17:43, Jacques Le Roux a écrit :
>> Le 17/11/2018 à 20:12, Mathieu Lirzin a écrit :
>>> For services I think the business logic and validation of data contained
>>> in services could often be extracted in separate unit testable methods
>>> which would allow cheap better case coverage to complement the
>>> integration test which is about checking that the service has the
>>> expected effect on the database.
>> Hi Mathieu,
>>
>> I see only one possible tiny drawback with this solution.
>>
>> I don't know you but most of the time, for small changes, I don't launch the integration tests, they are too long.
>> I wait for Buildbot feedback, which is most of the time not surprisingly good.
>>
>> Now every time we build OFBiz, the unit tests pass. Currently we have around thirty of them and they pass very quickly.
>> How many time it will take when we will have hundreds, if not thousands of them?
>>
>> Even if it's a small population, maybe we can already estimate an average time from the thirty we have?
>>
>> Thanks
>>
>> Jacques
>>
>>
> OK, I can answer to myself on this. This is not a problem at all.
> If someone thinks the test are too long in a particular circumstance
> s/he can use the same strategy that I often use for integration tests:
> skip them with "build -x test" and let Buildbot do the the work.

I think it is a good idea to run the unit tests before each commit which
is one of the reason why unit tests must be fast.

Currently we have 54 unit tests which takes 3+ seconds to compile and 7+
seconds to run on my machine.

To make this measurement I have deleted the ‘build’ directory.  Then I
have run the ‘classes’ task to compile every non test related classes.
I have measured the test compilation time with the ‘testClasses’ target
and test run time with the ‘test’ target.

Without considering the constant JVM warmup time and assuming linear
scalibity we could run about 400 unit tests in a minute.  However we
should expect better scalibity with modern multicore processors, since
unit tests are highly parallelizable because they don't rely on IO.  As
a consequence I bet we could have a few thousands of unit tests run in
less than a minute.

--
Mathieu Lirzin
GPG: F2A3 8D7E EB2B 6640 5761  070D 0ADE E100 9460 4D37