Importance of programming in our community

[taher]

Hello everyone,

I've been thinking for a few weeks about this topic, so I thought I'd
just share my thoughts here.

OFBiz has great DSL, it is a blessing, it makes things easier
OFBiz has great DSL, it is a curse, it makes things sloppy

I think OFBiz is unique and it's rare to find software projects where
non-programmers can participate. It makes the pool of participants
bigger. But there is a price to pay in terms of quality.

problem
-----------

OFBiz has too much DSL. The amount of XML in the code base makes
everything else looks tiny in comparison. As many programmers know, a
DSL (Domain Specific Language) is just a tool to make certain things
easier / faster. It is not _the_ solution for everything. In many
frameworks that I use when things get too complicated I drop down to
programming because the DSL hits certain limits and becomes a pain to
deal with.

When we write everything at a high level (DSL) we can accomplish a
lot, but the drawbacks include:
- Verbosity
- Copy-paste patterns
- Repetition due to inability to reuse
- Inflexible architecture:
  - services do not compose
  - debugging becomes hard
  - reusability is low
  - modularity is low

proposed direction
--------------------------

We have a word in the community: "Functional". We mean by this word a
person who is familiar with the entity model of OFBiz and understands
how the system works through the DSL but has limited or no ability to
program in a general purpose programming language. Many times when we
hit an architectural problem, a solution at the programming language
level can be as dramatic as 1% the size of the DSL solution and maybe
100 times faster. It can also be cleaner and more re-usable.

I think maybe part of the solution is to encourage some of our
"Functional" community members to embrace and enhance their
programming skills and not settle for only functional knowledge.
Examples for doing that:
- Embrace Java, it's the number 1 programming language on the planet
- Maybe attend basic coding lessons if possible
- Read online, plenty for resources
- Ask for help, from my experience people _love_ giving it here in the ASF
- Before writing a custom widget, think, maybe you can resolve it with
an FTL macro and make a root solution for many other widgets
- Before repeating a CRUD service for the 100th time, think, maybe you
can factor out common code into a nice Java service with parameters.
- Why not replace all this wiring of events, services and ECAs and
very complex route with a simple single service?

Do i make sense? Sorry if babbled a lot, and congratulations if you
made it to this line :) I'd love to hear your thoughts.

Cheers,

Taher Alkhateeb

[Pritam Kute]

Nice write-up Taher.

Yes, your thoughts are making sense to me. I would like to add one more
point from my side is, using DSL make learning curve steeper for the
newbies to the framework.

Thanks and Regards
--
Pritam Kute

On Mon, Sep 4, 2017 at 3:50 PM, Taher Alkhateeb <[hidden email]>
wrote:

> Hello everyone,
>
> I've been thinking for a few weeks about this topic, so I thought I'd
> just share my thoughts here.
>
> OFBiz has great DSL, it is a blessing, it makes things easier
> OFBiz has great DSL, it is a curse, it makes things sloppy
>
> I think OFBiz is unique and it's rare to find software projects where
> non-programmers can participate. It makes the pool of participants
> bigger. But there is a price to pay in terms of quality.
>
> problem
> -----------
>
> OFBiz has too much DSL. The amount of XML in the code base makes
> everything else looks tiny in comparison. As many programmers know, a
> DSL (Domain Specific Language) is just a tool to make certain things
> easier / faster. It is not _the_ solution for everything. In many
> frameworks that I use when things get too complicated I drop down to
> programming because the DSL hits certain limits and becomes a pain to
> deal with.
>
> When we write everything at a high level (DSL) we can accomplish a
> lot, but the drawbacks include:
> - Verbosity
> - Copy-paste patterns
> - Repetition due to inability to reuse
> - Inflexible architecture:
>   - services do not compose
>   - debugging becomes hard
>   - reusability is low
>   - modularity is low
>
> proposed direction
> --------------------------
>
> We have a word in the community: "Functional". We mean by this word a
> person who is familiar with the entity model of OFBiz and understands
> how the system works through the DSL but has limited or no ability to
> program in a general purpose programming language. Many times when we
> hit an architectural problem, a solution at the programming language
> level can be as dramatic as 1% the size of the DSL solution and maybe
> 100 times faster. It can also be cleaner and more re-usable.
>
> I think maybe part of the solution is to encourage some of our
> "Functional" community members to embrace and enhance their
> programming skills and not settle for only functional knowledge.
> Examples for doing that:
> - Embrace Java, it's the number 1 programming language on the planet
> - Maybe attend basic coding lessons if possible
> - Read online, plenty for resources
> - Ask for help, from my experience people _love_ giving it here in the ASF
> - Before writing a custom widget, think, maybe you can resolve it with
> an FTL macro and make a root solution for many other widgets
> - Before repeating a CRUD service for the 100th time, think, maybe you
> can factor out common code into a nice Java service with parameters.
> - Why not replace all this wiring of events, services and ECAs and
> very complex route with a simple single service?
>
> Do i make sense? Sorry if babbled a lot, and congratulations if you
> made it to this line :) I'd love to hear your thoughts.
>
> Cheers,
>
> Taher Alkhateeb
>

[Devanshu Vyas-2]

Hello Taher,

Thanks Taher for your thoughts, and I am in agreement with you. Writing
code in DSL make its easier for newbies but learning/understanding the
framework takes time and needs more care. We all should be working toward a
better framework with more generic purpose programming language use.

Thanks & Regards,
Devanshu Vyas.

On Mon, Sep 4, 2017 at 3:50 PM, Taher Alkhateeb <[hidden email]>
wrote:

> Hello everyone,
>
> I've been thinking for a few weeks about this topic, so I thought I'd
> just share my thoughts here.
>
> OFBiz has great DSL, it is a blessing, it makes things easier
> OFBiz has great DSL, it is a curse, it makes things sloppy
>
> I think OFBiz is unique and it's rare to find software projects where
> non-programmers can participate. It makes the pool of participants
> bigger. But there is a price to pay in terms of quality.
>
> problem
> -----------
>
> OFBiz has too much DSL. The amount of XML in the code base makes
> everything else looks tiny in comparison. As many programmers know, a
> DSL (Domain Specific Language) is just a tool to make certain things
> easier / faster. It is not _the_ solution for everything. In many
> frameworks that I use when things get too complicated I drop down to
> programming because the DSL hits certain limits and becomes a pain to
> deal with.
>
> When we write everything at a high level (DSL) we can accomplish a
> lot, but the drawbacks include:
> - Verbosity
> - Copy-paste patterns
> - Repetition due to inability to reuse
> - Inflexible architecture:
>   - services do not compose
>   - debugging becomes hard
>   - reusability is low
>   - modularity is low
>
> proposed direction
> --------------------------
>
> We have a word in the community: "Functional". We mean by this word a
> person who is familiar with the entity model of OFBiz and understands
> how the system works through the DSL but has limited or no ability to
> program in a general purpose programming language. Many times when we
> hit an architectural problem, a solution at the programming language
> level can be as dramatic as 1% the size of the DSL solution and maybe
> 100 times faster. It can also be cleaner and more re-usable.
>
> I think maybe part of the solution is to encourage some of our
> "Functional" community members to embrace and enhance their
> programming skills and not settle for only functional knowledge.
> Examples for doing that:
> - Embrace Java, it's the number 1 programming language on the planet
> - Maybe attend basic coding lessons if possible
> - Read online, plenty for resources
> - Ask for help, from my experience people _love_ giving it here in the ASF
> - Before writing a custom widget, think, maybe you can resolve it with
> an FTL macro and make a root solution for many other widgets
> - Before repeating a CRUD service for the 100th time, think, maybe you
> can factor out common code into a nice Java service with parameters.
> - Why not replace all this wiring of events, services and ECAs and
> very complex route with a simple single service?
>
> Do i make sense? Sorry if babbled a lot, and congratulations if you
> made it to this line :) I'd love to hear your thoughts.
>
> Cheers,
>
> Taher Alkhateeb
>

[Yash Sharma]

A big +1

On Mon, Sep 4, 2017 at 4:17 PM Taher Alkhateeb <[hidden email]>
wrote:

> Hello everyone,
>
> I've been thinking for a few weeks about this topic, so I thought I'd
> just share my thoughts here.
>
> OFBiz has great DSL, it is a blessing, it makes things easier
> OFBiz has great DSL, it is a curse, it makes things sloppy
>
> I think OFBiz is unique and it's rare to find software projects where
> non-programmers can participate. It makes the pool of participants
> bigger. But there is a price to pay in terms of quality.
>
> problem
> -----------
>
> OFBiz has too much DSL. The amount of XML in the code base makes
> everything else looks tiny in comparison. As many programmers know, a
> DSL (Domain Specific Language) is just a tool to make certain things
> easier / faster. It is not _the_ solution for everything. In many
> frameworks that I use when things get too complicated I drop down to
> programming because the DSL hits certain limits and becomes a pain to
> deal with.
>
> When we write everything at a high level (DSL) we can accomplish a
> lot, but the drawbacks include:
> - Verbosity
> - Copy-paste patterns
> - Repetition due to inability to reuse
> - Inflexible architecture:
>   - services do not compose
>   - debugging becomes hard
>   - reusability is low
>   - modularity is low
>
> proposed direction
> --------------------------
>
> We have a word in the community: "Functional". We mean by this word a
> person who is familiar with the entity model of OFBiz and understands
> how the system works through the DSL but has limited or no ability to
> program in a general purpose programming language. Many times when we
> hit an architectural problem, a solution at the programming language
> level can be as dramatic as 1% the size of the DSL solution and maybe
> 100 times faster. It can also be cleaner and more re-usable.
>
> I think maybe part of the solution is to encourage some of our
> "Functional" community members to embrace and enhance their
> programming skills and not settle for only functional knowledge.
> Examples for doing that:
> - Embrace Java, it's the number 1 programming language on the planet
> - Maybe attend basic coding lessons if possible
> - Read online, plenty for resources
> - Ask for help, from my experience people _love_ giving it here in the ASF
> - Before writing a custom widget, think, maybe you can resolve it with
> an FTL macro and make a root solution for many other widgets
> - Before repeating a CRUD service for the 100th time, think, maybe you
> can factor out common code into a nice Java service with parameters.
> - Why not replace all this wiring of events, services and ECAs and
> very complex route with a simple single service?
>
> Do i make sense? Sorry if babbled a lot, and congratulations if you
> made it to this line :) I'd love to hear your thoughts.
>
> Cheers,
>
> Taher Alkhateeb
>

--
Thanks,
*Pradhan Yash Sharma*

[Michael Brohl-3]

Hi Taher,

I agree with the general sight on these issues.

Regarding the point functionals vs. programmers: I'm not even sure if we
have a significant number of these people in the community? I guess most
of them are also programmers.

And if we have them, I think it will be hard to motivate functionals to
become programmers, at least for a significant number. It's already hard
to motivate programmers to participate in active development...


I think that we are going in the right direction with initiatives like
deprecating mini lang and converting it to Groovy. It takes a lot of
effort and time, though. With this effort should come a general solution
for CRUD services, or at least for some parts.

If you look at the minilang code, there is a copy&paste pattern to write
the created.. and lastUpdated.. fields. I think this is covered if we
use the entity engine Java code which automatically sets the values.

There are other parts, e.g. checking individual permissions for a
service, which cannot be generalized (or at least I have no solution in
mind yet). One example is the createProduct service, which does a few
checks on different permissions/roles.

But yes, we should think about reducing the code mass, avoid copy&paste
patterns and use centralized code as much as possible.


 > - Why not replace all this wiring of events, services and ECAs and
very complex route with a simple single service?

That's a point where we should be careful not to oversimplify things. It
might look complicated at first glance but the wiring gives you a lot of
flexibility which is really helpful in real customer projects.
Especially ECA's and SECA's are helping to prevent spaghetti code and
provide points in the logical chain where you can customize. You won't
be able to make a simple service out of this, it would be a huge service
with the loss of loose coupling.

In my eyes, a service should only be responsible for a single, limited
task. It's natural that you have a lot of services to chain together if
you want to make a complex business logic out of it.

Of course, this is generally spoken. There might be parts of the code
where we could simplify without loosing flexibility. But from my
experience, I would rather split up some huge Java service code (think
of ShoppingCart and Order functionality) and make it more flexible and
readable instead of making an even huger mess of code out of it.

My point here is that there is no general route to go. It depends on the
individual functionality and of the different quality we have in the code.


I think we should first focus on the ongoing initiatives to improve code
quality (minilang to Groovy conversion, FindBugs, etc.) and finish them.
It will be a huge effort until then.
After this, we are able to do next improvements on base of this code.

We can discuss, plan and design in the meantime, to be prepared for the
next steps.

Thanks and regards,
Michael



Am 04.09.17 um 12:20 schrieb Taher Alkhateeb:

> Hello everyone,
>
> I've been thinking for a few weeks about this topic, so I thought I'd
> just share my thoughts here.
>
> OFBiz has great DSL, it is a blessing, it makes things easier
> OFBiz has great DSL, it is a curse, it makes things sloppy
>
> I think OFBiz is unique and it's rare to find software projects where
> non-programmers can participate. It makes the pool of participants
> bigger. But there is a price to pay in terms of quality.
>
> problem
> -----------
>
> OFBiz has too much DSL. The amount of XML in the code base makes
> everything else looks tiny in comparison. As many programmers know, a
> DSL (Domain Specific Language) is just a tool to make certain things
> easier / faster. It is not _the_ solution for everything. In many
> frameworks that I use when things get too complicated I drop down to
> programming because the DSL hits certain limits and becomes a pain to
> deal with.
>
> When we write everything at a high level (DSL) we can accomplish a
> lot, but the drawbacks include:
> - Verbosity
> - Copy-paste patterns
> - Repetition due to inability to reuse
> - Inflexible architecture:
>    - services do not compose
>    - debugging becomes hard
>    - reusability is low
>    - modularity is low
>
> proposed direction
> --------------------------
>
> We have a word in the community: "Functional". We mean by this word a
> person who is familiar with the entity model of OFBiz and understands
> how the system works through the DSL but has limited or no ability to
> program in a general purpose programming language. Many times when we
> hit an architectural problem, a solution at the programming language
> level can be as dramatic as 1% the size of the DSL solution and maybe
> 100 times faster. It can also be cleaner and more re-usable.
>
> I think maybe part of the solution is to encourage some of our
> "Functional" community members to embrace and enhance their
> programming skills and not settle for only functional knowledge.
> Examples for doing that:
> - Embrace Java, it's the number 1 programming language on the planet
> - Maybe attend basic coding lessons if possible
> - Read online, plenty for resources
> - Ask for help, from my experience people _love_ giving it here in the ASF
> - Before writing a custom widget, think, maybe you can resolve it with
> an FTL macro and make a root solution for many other widgets
> - Before repeating a CRUD service for the 100th time, think, maybe you
> can factor out common code into a nice Java service with parameters.
> - Why not replace all this wiring of events, services and ECAs and
> very complex route with a simple single service?
>
> Do i make sense? Sorry if babbled a lot, and congratulations if you
> made it to this line :) I'd love to hear your thoughts.
>
> Cheers,
>
> Taher Alkhateeb

[Jacques Le Roux]

Hi Michael, everyone concerned,


Le 10/09/2017 à 11:41, Michael Brohl a écrit :

> Hi Taher,
>
> I agree with the general sight on these issues.
>
> Regarding the point functionals vs. programmers: I'm not even sure if we have a significant number of these people in the community? I guess most of
> them are also programmers.
>
> And if we have them, I think it will be hard to motivate functionals to become programmers, at least for a significant number. It's already hard to
> motivate programmers to participate in active development...
>
>
> I think that we are going in the right direction with initiatives like deprecating mini lang and converting it to Groovy. It takes a lot of effort
> and time, though. With this effort should come a general solution for CRUD services, or at least for some parts.

With  Nicolas's effort in 2016 and this year and the great help from the community this year we have already covered a large part, kudos to everyone
who participated

>
> If you look at the minilang code, there is a copy&paste pattern to write the created.. and lastUpdated.. fields. I think this is covered if we use
> the entity engine Java code which automatically sets the values.
We can maybe indeed automate things more, did not think about it

>
> There are other parts, e.g. checking individual permissions for a service, which cannot be generalized (or at least I have no solution in mind yet).
> One example is the createProduct service, which does a few checks on different permissions/roles.
>
> But yes, we should think about reducing the code mass, avoid copy&paste patterns and use centralized code as much as possible.
+1

>
> > - Why not replace all this wiring of events, services and ECAs and very complex route with a simple single service?
>
> That's a point where we should be careful not to oversimplify things. It might look complicated at first glance but the wiring gives you a lot of
> flexibility which is really helpful in real customer projects. Especially ECA's and SECA's are helping to prevent spaghetti code and provide points
> in the logical chain where you can customize. You won't be able to make a simple service out of this, it would be a huge service with the loss of
> loose coupling.
I totally agree for  ECAs. They are used in OFBiz to handle complex workflow. And even pragmatically replaced the initially used workflow engine.

> In my eyes, a service should only be responsible for a single, limited task. It's natural that you have a lot of services to chain together if you
> want to make a complex business logic out of it.
I agree and it's certainly possible to break down complex services. A service should be on tackling a simple task. You can even group services
routinely used together in a sequence using the group service engine

>
> Of course, this is generally spoken. There might be parts of the code where we could simplify without loosing flexibility. But from my experience, I
> would rather split up some huge Java service code (think of ShoppingCart and Order functionality) and make it more flexible and readable instead of
> making an even huger mess of code out of it.
+1

>
> My point here is that there is no general route to go. It depends on the individual functionality and of the different quality we have in the code.
>
>
> I think we should first focus on the ongoing initiatives to improve code quality (minilang to Groovy conversion, FindBugs, etc.) and finish them. It
> will be a huge effort until then.
> After this, we are able to do next improvements on base of this code.
I already mentioned that we should be very careful when replacing Minilang code and we should be sure that every replacement is covered by a test
before replacing it.

Jacques

>
> We can discuss, plan and design in the meantime, to be prepared for the next steps.
>
> Thanks and regards,
> Michael
>
>
>
> Am 04.09.17 um 12:20 schrieb Taher Alkhateeb:
>> Hello everyone,
>>
>> I've been thinking for a few weeks about this topic, so I thought I'd
>> just share my thoughts here.
>>
>> OFBiz has great DSL, it is a blessing, it makes things easier
>> OFBiz has great DSL, it is a curse, it makes things sloppy
>>
>> I think OFBiz is unique and it's rare to find software projects where
>> non-programmers can participate. It makes the pool of participants
>> bigger. But there is a price to pay in terms of quality.
>>
>> problem
>> -----------
>>
>> OFBiz has too much DSL. The amount of XML in the code base makes
>> everything else looks tiny in comparison. As many programmers know, a
>> DSL (Domain Specific Language) is just a tool to make certain things
>> easier / faster. It is not _the_ solution for everything. In many
>> frameworks that I use when things get too complicated I drop down to
>> programming because the DSL hits certain limits and becomes a pain to
>> deal with.
>>
>> When we write everything at a high level (DSL) we can accomplish a
>> lot, but the drawbacks include:
>> - Verbosity
>> - Copy-paste patterns
>> - Repetition due to inability to reuse
>> - Inflexible architecture:
>>    - services do not compose
>>    - debugging becomes hard
>>    - reusability is low
>>    - modularity is low
>>
>> proposed direction
>> --------------------------
>>
>> We have a word in the community: "Functional". We mean by this word a
>> person who is familiar with the entity model of OFBiz and understands
>> how the system works through the DSL but has limited or no ability to
>> program in a general purpose programming language. Many times when we
>> hit an architectural problem, a solution at the programming language
>> level can be as dramatic as 1% the size of the DSL solution and maybe
>> 100 times faster. It can also be cleaner and more re-usable.
>>
>> I think maybe part of the solution is to encourage some of our
>> "Functional" community members to embrace and enhance their
>> programming skills and not settle for only functional knowledge.
>> Examples for doing that:
>> - Embrace Java, it's the number 1 programming language on the planet
>> - Maybe attend basic coding lessons if possible
>> - Read online, plenty for resources
>> - Ask for help, from my experience people _love_ giving it here in the ASF
>> - Before writing a custom widget, think, maybe you can resolve it with
>> an FTL macro and make a root solution for many other widgets
>> - Before repeating a CRUD service for the 100th time, think, maybe you
>> can factor out common code into a nice Java service with parameters.
>> - Why not replace all this wiring of events, services and ECAs and
>> very complex route with a simple single service?
>>
>> Do i make sense? Sorry if babbled a lot, and congratulations if you
>> made it to this line :) I'd love to hear your thoughts.
>>
>> Cheers,
>>
>> Taher Alkhateeb
>
>