Tracking Data Model changes

[Aditya Sharma]

Hello everyone,

For one of my assignments, I need to find out entity changes that took
place between an older release and the latest release.

One of the solutions that came up was comparing the database using MySQL
Workbench or some other utility. I found around 60+ new entity changes and
a lot of minor field changes since last big book was published (OFBiz 9 I
suppose).
It's fascinating for me that around 8 years passed since then and data
model still stands well (Kudos to Universal Data Model that we followed in
OFBiz)


Just a proposal, since we don't have so many frequent changes in the data
model. It will be good to have a page or some other method defined to keep
a track of such changes.

I feel such information can be quite helpful when migrating from an older
to some newer release.

Please share your thoughts.

Thanks and Regards,

*Aditya Sharma* | Enterprise Software Engineer
HotWax Systems <http://www.hotwaxsystems.com/>
<https://www.linkedin.com/in/aditya-sharma-78291810a/>

[Jacques Le Roux]

+1 clearly

Jacques


Le 30/08/2017 à 15:23, Aditya Sharma a écrit :

> Hello everyone,
>
> For one of my assignments, I need to find out entity changes that took
> place between an older release and the latest release.
>
> One of the solutions that came up was comparing the database using MySQL
> Workbench or some other utility. I found around 60+ new entity changes and
> a lot of minor field changes since last big book was published (OFBiz 9 I
> suppose).
> It's fascinating for me that around 8 years passed since then and data
> model still stands well (Kudos to Universal Data Model that we followed in
> OFBiz)
>
>
> Just a proposal, since we don't have so many frequent changes in the data
> model. It will be good to have a page or some other method defined to keep
> a track of such changes.
>
> I feel such information can be quite helpful when migrating from an older
> to some newer release.
>
> Please share your thoughts.
>
> Thanks and Regards,
>
> *Aditya Sharma* | Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>

[Aishwary Shrivastava]

Indeed it's a great idea, Aditya!!
+1

On Wed, Aug 30, 2017 at 7:06 PM, Jacques Le Roux <
[hidden email]> wrote:

> +1 clearly
>
> Jacques
>
>
>
> Le 30/08/2017 à 15:23, Aditya Sharma a écrit :
>
>> Hello everyone,
>>
>> For one of my assignments, I need to find out entity changes that took
>> place between an older release and the latest release.
>>
>> One of the solutions that came up was comparing the database using MySQL
>> Workbench or some other utility. I found around 60+ new entity changes and
>> a lot of minor field changes since last big book was published (OFBiz 9 I
>> suppose).
>> It's fascinating for me that around 8 years passed since then and data
>> model still stands well (Kudos to Universal Data Model that we followed in
>> OFBiz)
>>
>>
>> Just a proposal, since we don't have so many frequent changes in the data
>> model. It will be good to have a page or some other method defined to keep
>> a track of such changes.
>>
>> I feel such information can be quite helpful when migrating from an older
>> to some newer release.
>>
>> Please share your thoughts.
>>
>> Thanks and Regards,
>>
>> *Aditya Sharma* | Enterprise Software Engineer
>> HotWax Systems <http://www.hotwaxsystems.com/>
>> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>>
>>
>

--
Best regards,
Aishwary Shrivastava
Enterprise Software Engineer
HotWax Systems <http://www.hotwaxsystems.com/>

[Lalit Dashora]

+1 Great Idea Aditya !!!

On Wed, Aug 30, 2017 at 7:28 PM, Aishwary Shrivastava <
[hidden email]> wrote:

> Indeed it's a great idea, Aditya!!
> +1
>
> On Wed, Aug 30, 2017 at 7:06 PM, Jacques Le Roux <
> [hidden email]> wrote:
>
> > +1 clearly
> >
> > Jacques
> >
> >
> >
> > Le 30/08/2017 à 15:23, Aditya Sharma a écrit :
> >
> >> Hello everyone,
> >>
> >> For one of my assignments, I need to find out entity changes that took
> >> place between an older release and the latest release.
> >>
> >> One of the solutions that came up was comparing the database using MySQL
> >> Workbench or some other utility. I found around 60+ new entity changes
> and
> >> a lot of minor field changes since last big book was published (OFBiz 9
> I
> >> suppose).
> >> It's fascinating for me that around 8 years passed since then and data
> >> model still stands well (Kudos to Universal Data Model that we followed
> in
> >> OFBiz)
> >>
> >>
> >> Just a proposal, since we don't have so many frequent changes in the
> data
> >> model. It will be good to have a page or some other method defined to
> keep
> >> a track of such changes.
> >>
> >> I feel such information can be quite helpful when migrating from an
> older
> >> to some newer release.
> >>
> >> Please share your thoughts.
> >>
> >> Thanks and Regards,
> >>
> >> *Aditya Sharma* | Enterprise Software Engineer
> >> HotWax Systems <http://www.hotwaxsystems.com/>
> >> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
> >>
> >>
> >
>
>
> --
> Best regards,
> Aishwary Shrivastava
> Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
>

--
Lalit Dashora
Enterprise Software Engineer

Hotwax Systems

[taher]

Good idea! Why not take it a step further, and write data migration
scripts? They will serve two purposes in one: 1) document changes 2)
automate upgrades.

You can experiment with solutions like liquibase or flyway

On Aug 30, 2017 4:23 PM, "Aditya Sharma" <[hidden email]>
wrote:

Hello everyone,

For one of my assignments, I need to find out entity changes that took
place between an older release and the latest release.

One of the solutions that came up was comparing the database using MySQL
Workbench or some other utility. I found around 60+ new entity changes and
a lot of minor field changes since last big book was published (OFBiz 9 I
suppose).
It's fascinating for me that around 8 years passed since then and data
model still stands well (Kudos to Universal Data Model that we followed in
OFBiz)


Just a proposal, since we don't have so many frequent changes in the data
model. It will be good to have a page or some other method defined to keep
a track of such changes.

I feel such information can be quite helpful when migrating from an older
to some newer release.

Please share your thoughts.

Thanks and Regards,

*Aditya Sharma* | Enterprise Software Engineer
HotWax Systems <http://www.hotwaxsystems.com/>
<https://www.linkedin.com/in/aditya-sharma-78291810a/>

[Jacques Le Roux]

Hi Taher,

You mean something more than https://cwiki.apache.org/confluence/display/OFBIZ/Revisions+Requiring+Data+Migration+-+upgrade+ofbiz ?

If yes, why?

Thanks

Jacques


Le 30/08/2017 à 16:24, Taher Alkhateeb a écrit :

> Good idea! Why not take it a step further, and write data migration
> scripts? They will serve two purposes in one: 1) document changes 2)
> automate upgrades.
>
> You can experiment with solutions like liquibase or flyway
>
> On Aug 30, 2017 4:23 PM, "Aditya Sharma" <[hidden email]>
> wrote:
>
> Hello everyone,
>
> For one of my assignments, I need to find out entity changes that took
> place between an older release and the latest release.
>
> One of the solutions that came up was comparing the database using MySQL
> Workbench or some other utility. I found around 60+ new entity changes and
> a lot of minor field changes since last big book was published (OFBiz 9 I
> suppose).
> It's fascinating for me that around 8 years passed since then and data
> model still stands well (Kudos to Universal Data Model that we followed in
> OFBiz)
>
>
> Just a proposal, since we don't have so many frequent changes in the data
> model. It will be good to have a page or some other method defined to keep
> a track of such changes.
>
> I feel such information can be quite helpful when migrating from an older
> to some newer release.
>
> Please share your thoughts.
>
> Thanks and Regards,
>
> *Aditya Sharma* | Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>

[Ashish Vijaywargiya-4]

+1, Taher. I will wait for your comment on Jacques question, we already
have a document but I think the automated script that can be implemented
here. Liquidbase and flyway seem to be promising solutions!

One question always comes to my mind: Can we say that automated scripts
will support the migration from last two or at max three known releases?
I think we should not put the effort in building the migration script that
could migrate ofbiz earlier version(Let's say Ofbiz 10 or 9 or 4) to the
latest version. Please share your thoughts on this.

Kind Regards
Ashish Vijaywargiya
HotWax Systems - est. 1997 <http://www.hotwaxsystems.com/>


On Wed, Aug 30, 2017 at 7:54 PM, Taher Alkhateeb <[hidden email]
> wrote:

> Good idea! Why not take it a step further, and write data migration
> scripts? They will serve two purposes in one: 1) document changes 2)
> automate upgrades.
>
> You can experiment with solutions like liquibase or flyway
>
> On Aug 30, 2017 4:23 PM, "Aditya Sharma" <[hidden email]>
> wrote:
>
> Hello everyone,
>
> For one of my assignments, I need to find out entity changes that took
> place between an older release and the latest release.
>
> One of the solutions that came up was comparing the database using MySQL
> Workbench or some other utility. I found around 60+ new entity changes and
> a lot of minor field changes since last big book was published (OFBiz 9 I
> suppose).
> It's fascinating for me that around 8 years passed since then and data
> model still stands well (Kudos to Universal Data Model that we followed in
> OFBiz)
>
>
> Just a proposal, since we don't have so many frequent changes in the data
> model. It will be good to have a page or some other method defined to keep
> a track of such changes.
>
> I feel such information can be quite helpful when migrating from an older
> to some newer release.
>
> Please share your thoughts.
>
> Thanks and Regards,
>
> *Aditya Sharma* | Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>

[Pranay Pandey-3]

It's a good idea +1.

Best regards,
Pranay Pandey
www.hotwaxsystems.com
www.hotwax.co

On Wed, Aug 30, 2017 at 6:53 PM, Aditya Sharma <
[hidden email]> wrote:

> Hello everyone,
>
> For one of my assignments, I need to find out entity changes that took
> place between an older release and the latest release.
>
> One of the solutions that came up was comparing the database using MySQL
> Workbench or some other utility. I found around 60+ new entity changes and
> a lot of minor field changes since last big book was published (OFBiz 9 I
> suppose).
> It's fascinating for me that around 8 years passed since then and data
> model still stands well (Kudos to Universal Data Model that we followed in
> OFBiz)
>
>
> Just a proposal, since we don't have so many frequent changes in the data
> model. It will be good to have a page or some other method defined to keep
> a track of such changes.
>
> I feel such information can be quite helpful when migrating from an older
> to some newer release.
>
> Please share your thoughts.
>
> Thanks and Regards,
>
> *Aditya Sharma* | Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>

[taher]

Hi Ashish,

With respect to Jacques' question, I kind of already answered in that
it is not only documentation but also automation.

Now with respect to which releases to incorporate, it really depends
on what the community decides. I would personally prefer to not go any
earlier than 13, or preferably just 16 to trunk, which means we design
this solution for the future, not necessarily the past. The powerful
thing in using something like liquibase is that not only do you change
the schema (the entity engine can do that partially) but you also
decide on how to migrate the existing data to the new schema. For
example, you might need to split a field to two, or merge two fields,
and so on and so forth.

Anyway, this is only an idea if people are interested in it. The
original idea of just documenting is also perfectly reasonable and
beneficial.

On Thu, Aug 31, 2017 at 8:11 AM, Ashish Vijaywargiya
<[hidden email]> wrote:

> +1, Taher. I will wait for your comment on Jacques question, we already
> have a document but I think the automated script that can be implemented
> here. Liquidbase and flyway seem to be promising solutions!
>
> One question always comes to my mind: Can we say that automated scripts
> will support the migration from last two or at max three known releases?
> I think we should not put the effort in building the migration script that
> could migrate ofbiz earlier version(Let's say Ofbiz 10 or 9 or 4) to the
> latest version. Please share your thoughts on this.
>
> Kind Regards
> Ashish Vijaywargiya
> HotWax Systems - est. 1997 <http://www.hotwaxsystems.com/>
>
>
> On Wed, Aug 30, 2017 at 7:54 PM, Taher Alkhateeb <[hidden email]
>> wrote:
>
>> Good idea! Why not take it a step further, and write data migration
>> scripts? They will serve two purposes in one: 1) document changes 2)
>> automate upgrades.
>>
>> You can experiment with solutions like liquibase or flyway
>>
>> On Aug 30, 2017 4:23 PM, "Aditya Sharma" <[hidden email]>
>> wrote:
>>
>> Hello everyone,
>>
>> For one of my assignments, I need to find out entity changes that took
>> place between an older release and the latest release.
>>
>> One of the solutions that came up was comparing the database using MySQL
>> Workbench or some other utility. I found around 60+ new entity changes and
>> a lot of minor field changes since last big book was published (OFBiz 9 I
>> suppose).
>> It's fascinating for me that around 8 years passed since then and data
>> model still stands well (Kudos to Universal Data Model that we followed in
>> OFBiz)
>>
>>
>> Just a proposal, since we don't have so many frequent changes in the data
>> model. It will be good to have a page or some other method defined to keep
>> a track of such changes.
>>
>> I feel such information can be quite helpful when migrating from an older
>> to some newer release.
>>
>> Please share your thoughts.
>>
>> Thanks and Regards,
>>
>> *Aditya Sharma* | Enterprise Software Engineer
>> HotWax Systems <http://www.hotwaxsystems.com/>
>> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>>

[Jacques Le Roux]

Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
> I would personally prefer to not go any
> earlier than 13, or preferably just 16 to trunk, which means we design
> this solution for the future, not necessarily the past.
+1

Jacques

[Nicolas Malin-2]

I'm in favor to tracking the different migration but at this time I
didn't see the advantage to use flyway or other instead of manage easily
by ofbiz throw groovy script.

I'm available to help for design or create a POC do realize it because
many time in the past (and currently ow) I want to refactoring some
code/db schema with data migration but we haven't clean process to do that.

Nicolas

Le 31/08/2017 à 12:37, Jacques Le Roux a écrit :
> Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
>> I would personally prefer to not go any
>> earlier than 13, or preferably just 16 to trunk, which means we design
>> this solution for the future, not necessarily the past.
> +1
>
> Jacques
>
>

[taher]

Groovy scripts are also great and can do the job. To comment on the
"advantage" of something like flyway or liquibase I can try to list
some:
- The scripts might get too big or complex to accommodate different
databases and platforms.
- Out of the box, these solutions are database independent
- Ability to redo / undo on schema changes
- Supporting declarative style for schema definitions based on
multiple formats (YAML, XML, JSON, etc ..)
- The DSL is easier to use (declarative and short)

So in short, both solutions are viable, and existing solutions might
be a bit easier to implement especially if you consider additional
features in those solutions.

On Fri, Sep 1, 2017 at 2:23 PM, Nicolas Malin <[hidden email]> wrote:

> I'm in favor to tracking the different migration but at this time I didn't
> see the advantage to use flyway or other instead of manage easily by ofbiz
> throw groovy script.
>
> I'm available to help for design or create a POC do realize it because many
> time in the past (and currently ow) I want to refactoring some code/db
> schema with data migration but we haven't clean process to do that.
>
> Nicolas
>
>
> Le 31/08/2017 à 12:37, Jacques Le Roux a écrit :
>>
>> Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
>>>
>>> I would personally prefer to not go any
>>> earlier than 13, or preferably just 16 to trunk, which means we design
>>> this solution for the future, not necessarily the past.
>>
>> +1
>>
>> Jacques
>>
>>
>

[Michael Brohl-3]

+1 for the documentation of database changes.

We alreadyy do this for customer projects, along with (database
specific) data migration scripts.

I'm not sure if we can afford to provide sophisticated additional tool
support which is maintained continiously?


As a conclusion, I think we should setup a convention that any database
model change has to provide a proper change log and migration script for
the embedded database (if applicable).

Thanks,

Michael


Am 01.09.17 um 15:24 schrieb Taher Alkhateeb:

> Groovy scripts are also great and can do the job. To comment on the
> "advantage" of something like flyway or liquibase I can try to list
> some:
> - The scripts might get too big or complex to accommodate different
> databases and platforms.
> - Out of the box, these solutions are database independent
> - Ability to redo / undo on schema changes
> - Supporting declarative style for schema definitions based on
> multiple formats (YAML, XML, JSON, etc ..)
> - The DSL is easier to use (declarative and short)
>
> So in short, both solutions are viable, and existing solutions might
> be a bit easier to implement especially if you consider additional
> features in those solutions.
>
> On Fri, Sep 1, 2017 at 2:23 PM, Nicolas Malin <[hidden email]> wrote:
>> I'm in favor to tracking the different migration but at this time I didn't
>> see the advantage to use flyway or other instead of manage easily by ofbiz
>> throw groovy script.
>>
>> I'm available to help for design or create a POC do realize it because many
>> time in the past (and currently ow) I want to refactoring some code/db
>> schema with data migration but we haven't clean process to do that.
>>
>> Nicolas
>>
>>
>> Le 31/08/2017 à 12:37, Jacques Le Roux a écrit :
>>> Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
>>>> I would personally prefer to not go any
>>>> earlier than 13, or preferably just 16 to trunk, which means we design
>>>> this solution for the future, not necessarily the past.
>>> +1
>>>
>>> Jacques
>>>
>>>

[Aditya Sharma]

I see two ways to do that

1. A page maintained on Confluence where all the data model changes are
updated. Here, we can have a page maintained for the upcoming release when
the release is out we make it sub child titled with the release name.
2. A file is maintained in ofbiz-framework code base itself that goes with
the package with information about data model changes. Whenever someone
downloads the package this file will help track data model changes that
come with the package.

We can make it part of contributor's practice to update it whenever such
changes are done.

Which way should we go?

Thanks and Regards,

*Aditya Sharma* | Enterprise Software Engineer
HotWax Systems <http://www.hotwaxsystems.com/>
<https://www.linkedin.com/in/aditya-sharma-78291810a/>
On Sun, Sep 10, 2017 at 4:46 PM, Michael Brohl <[hidden email]>
wrote:

> +1 for the documentation of database changes.
>
> We alreadyy do this for customer projects, along with (database specific)
> data migration scripts.
>
> I'm not sure if we can afford to provide sophisticated additional tool
> support which is maintained continiously?
>
>
> As a conclusion, I think we should setup a convention that any database
> model change has to provide a proper change log and migration script for
> the embedded database (if applicable).
>
> Thanks,
>
> Michael
>
>
> Am 01.09.17 um 15:24 schrieb Taher Alkhateeb:
>
> Groovy scripts are also great and can do the job. To comment on the
>> "advantage" of something like flyway or liquibase I can try to list
>> some:
>> - The scripts might get too big or complex to accommodate different
>> databases and platforms.
>> - Out of the box, these solutions are database independent
>> - Ability to redo / undo on schema changes
>> - Supporting declarative style for schema definitions based on
>> multiple formats (YAML, XML, JSON, etc ..)
>> - The DSL is easier to use (declarative and short)
>>
>> So in short, both solutions are viable, and existing solutions might
>> be a bit easier to implement especially if you consider additional
>> features in those solutions.
>>
>> On Fri, Sep 1, 2017 at 2:23 PM, Nicolas Malin <[hidden email]>
>> wrote:
>>
>>> I'm in favor to tracking the different migration but at this time I
>>> didn't
>>> see the advantage to use flyway or other instead of manage easily by
>>> ofbiz
>>> throw groovy script.
>>>
>>> I'm available to help for design or create a POC do realize it because
>>> many
>>> time in the past (and currently ow) I want to refactoring some code/db
>>> schema with data migration but we haven't clean process to do that.
>>>
>>> Nicolas
>>>
>>>
>>> Le 31/08/2017 à 12:37, Jacques Le Roux a écrit :
>>>
>>>> Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
>>>>
>>>>> I would personally prefer to not go any
>>>>> earlier than 13, or preferably just 16 to trunk, which means we design
>>>>> this solution for the future, not necessarily the past.
>>>>>
>>>> +1
>>>>
>>>> Jacques
>>>>
>>>>
>>>>
>
>

[Jacques Le Roux]

Hi Aditya,

One way we could use and is already used for the Gradle and Birt Flexible documentation in wiki is to create README.md files, uses Pandoc to generate
a HTML file from it in tools\wiki-files and import this file in wiki using the HTML import macro.

Doing  so we follow both way you suggested. So users have it in 2 places while it's only maintained in one place where it's versioned (though
Confluence also versions pages)

Maybe an overkill in this case though. Since we have already
https://cwiki.apache.org/confluence/display/OFBIZ/Revisions+Requiring+Data+Migration+-+upgrade+ofbiz to start from

My 2cts

Jacques


Le 23/09/2017 à 10:41, Aditya Sharma a écrit :

> I see two ways to do that
>
> 1. A page maintained on Confluence where all the data model changes are
> updated. Here, we can have a page maintained for the upcoming release when
> the release is out we make it sub child titled with the release name.
> 2. A file is maintained in ofbiz-framework code base itself that goes with
> the package with information about data model changes. Whenever someone
> downloads the package this file will help track data model changes that
> come with the package.
>
> We can make it part of contributor's practice to update it whenever such
> changes are done.
>
> Which way should we go?
>
> Thanks and Regards,
>
> *Aditya Sharma* | Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
> On Sun, Sep 10, 2017 at 4:46 PM, Michael Brohl <[hidden email]>
> wrote:
>
>> +1 for the documentation of database changes.
>>
>> We alreadyy do this for customer projects, along with (database specific)
>> data migration scripts.
>>
>> I'm not sure if we can afford to provide sophisticated additional tool
>> support which is maintained continiously?
>>
>>
>> As a conclusion, I think we should setup a convention that any database
>> model change has to provide a proper change log and migration script for
>> the embedded database (if applicable).
>>
>> Thanks,
>>
>> Michael
>>
>>
>> Am 01.09.17 um 15:24 schrieb Taher Alkhateeb:
>>
>> Groovy scripts are also great and can do the job. To comment on the
>>> "advantage" of something like flyway or liquibase I can try to list
>>> some:
>>> - The scripts might get too big or complex to accommodate different
>>> databases and platforms.
>>> - Out of the box, these solutions are database independent
>>> - Ability to redo / undo on schema changes
>>> - Supporting declarative style for schema definitions based on
>>> multiple formats (YAML, XML, JSON, etc ..)
>>> - The DSL is easier to use (declarative and short)
>>>
>>> So in short, both solutions are viable, and existing solutions might
>>> be a bit easier to implement especially if you consider additional
>>> features in those solutions.
>>>
>>> On Fri, Sep 1, 2017 at 2:23 PM, Nicolas Malin <[hidden email]>
>>> wrote:
>>>
>>>> I'm in favor to tracking the different migration but at this time I
>>>> didn't
>>>> see the advantage to use flyway or other instead of manage easily by
>>>> ofbiz
>>>> throw groovy script.
>>>>
>>>> I'm available to help for design or create a POC do realize it because
>>>> many
>>>> time in the past (and currently ow) I want to refactoring some code/db
>>>> schema with data migration but we haven't clean process to do that.
>>>>
>>>> Nicolas
>>>>
>>>>
>>>> Le 31/08/2017 à 12:37, Jacques Le Roux a écrit :
>>>>
>>>>> Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
>>>>>
>>>>>> I would personally prefer to not go any
>>>>>> earlier than 13, or preferably just 16 to trunk, which means we design
>>>>>> this solution for the future, not necessarily the past.
>>>>>>
>>>>> +1
>>>>>
>>>>> Jacques
>>>>>
>>>>>
>>>>>
>>

[Aditya Sharma]

That would be a much effective solution.

As far as https://cwiki.apache.org/confluence/display/OFBIZ/
Revisions+Requiring+Data+Migration+-+upgrade+ofbiz page is concerned I find
it quite absurd that we provide information based upon only revisions while
users deal with releases.

Though it would be better to link it to data migration page so that user
gets all information through a single path.

Thanks and Regards,

*Aditya Sharma* | Enterprise Software Engineer
HotWax Systems <http://www.hotwaxsystems.com/>
<https://www.linkedin.com/in/aditya-sharma-78291810a/>

On Sat, Sep 23, 2017 at 2:22 PM, Jacques Le Roux <
[hidden email]> wrote:

> Hi Aditya,
>
> One way we could use and is already used for the Gradle and Birt Flexible
> documentation in wiki is to create README.md files, uses Pandoc to generate
> a HTML file from it in tools\wiki-files and import this file in wiki using
> the HTML import macro.
>
> Doing  so we follow both way you suggested. So users have it in 2 places
> while it's only maintained in one place where it's versioned (though
> Confluence also versions pages)
>
> Maybe an overkill in this case though. Since we have already
> https://cwiki.apache.org/confluence/display/OFBIZ/Revisions+
> Requiring+Data+Migration+-+upgrade+ofbiz to start from
>
> My 2cts
>
> Jacques
>
>
> Le 23/09/2017 à 10:41, Aditya Sharma a écrit :
>
>> I see two ways to do that
>>
>> 1. A page maintained on Confluence where all the data model changes are
>> updated. Here, we can have a page maintained for the upcoming release when
>> the release is out we make it sub child titled with the release name.
>> 2. A file is maintained in ofbiz-framework code base itself that goes with
>> the package with information about data model changes. Whenever someone
>> downloads the package this file will help track data model changes that
>> come with the package.
>>
>> We can make it part of contributor's practice to update it whenever such
>> changes are done.
>>
>> Which way should we go?
>>
>> Thanks and Regards,
>>
>> *Aditya Sharma* | Enterprise Software Engineer
>> HotWax Systems <http://www.hotwaxsystems.com/>
>> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>>
>> On Sun, Sep 10, 2017 at 4:46 PM, Michael Brohl <[hidden email]>
>> wrote:
>>
>> +1 for the documentation of database changes.
>>>
>>> We alreadyy do this for customer projects, along with (database specific)
>>> data migration scripts.
>>>
>>> I'm not sure if we can afford to provide sophisticated additional tool
>>> support which is maintained continiously?
>>>
>>>
>>> As a conclusion, I think we should setup a convention that any database
>>> model change has to provide a proper change log and migration script for
>>> the embedded database (if applicable).
>>>
>>> Thanks,
>>>
>>> Michael
>>>
>>>
>>> Am 01.09.17 um 15:24 schrieb Taher Alkhateeb:
>>>
>>> Groovy scripts are also great and can do the job. To comment on the
>>>
>>>> "advantage" of something like flyway or liquibase I can try to list
>>>> some:
>>>> - The scripts might get too big or complex to accommodate different
>>>> databases and platforms.
>>>> - Out of the box, these solutions are database independent
>>>> - Ability to redo / undo on schema changes
>>>> - Supporting declarative style for schema definitions based on
>>>> multiple formats (YAML, XML, JSON, etc ..)
>>>> - The DSL is easier to use (declarative and short)
>>>>
>>>> So in short, both solutions are viable, and existing solutions might
>>>> be a bit easier to implement especially if you consider additional
>>>> features in those solutions.
>>>>
>>>> On Fri, Sep 1, 2017 at 2:23 PM, Nicolas Malin <[hidden email]
>>>> >
>>>> wrote:
>>>>
>>>> I'm in favor to tracking the different migration but at this time I
>>>>> didn't
>>>>> see the advantage to use flyway or other instead of manage easily by
>>>>> ofbiz
>>>>> throw groovy script.
>>>>>
>>>>> I'm available to help for design or create a POC do realize it because
>>>>> many
>>>>> time in the past (and currently ow) I want to refactoring some code/db
>>>>> schema with data migration but we haven't clean process to do that.
>>>>>
>>>>> Nicolas
>>>>>
>>>>>
>>>>> Le 31/08/2017 à 12:37, Jacques Le Roux a écrit :
>>>>>
>>>>> Le 31/08/2017 à 12:32, Taher Alkhateeb a écrit :
>>>>>>
>>>>>> I would personally prefer to not go any
>>>>>>> earlier than 13, or preferably just 16 to trunk, which means we
>>>>>>> design
>>>>>>> this solution for the future, not necessarily the past.
>>>>>>>
>>>>>>> +1
>>>>>>
>>>>>> Jacques
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>

[Michael Brohl-3]

+1 for tracking datamodel changes together with data migration scripts

In our customer projects, we track every change in a simple text file in
the source code repository. It contains description of the changes,
references to issues or requirement documentation and SQL scripts for
migrations.

In OFBiz, we should at least provide SQL scripts for the embedded Derby
database, maybe there will be contributions for other databases as well.

Something like db-changelog.derby.txt, db-changelog.mysql.txt or similar.

Cheers,

Michael

Am 31.08.17 um 12:32 schrieb Taher Alkhateeb:

> Hi Ashish,
>
> With respect to Jacques' question, I kind of already answered in that
> it is not only documentation but also automation.
>
> Now with respect to which releases to incorporate, it really depends
> on what the community decides. I would personally prefer to not go any
> earlier than 13, or preferably just 16 to trunk, which means we design
> this solution for the future, not necessarily the past. The powerful
> thing in using something like liquibase is that not only do you change
> the schema (the entity engine can do that partially) but you also
> decide on how to migrate the existing data to the new schema. For
> example, you might need to split a field to two, or merge two fields,
> and so on and so forth.
>
> Anyway, this is only an idea if people are interested in it. The
> original idea of just documenting is also perfectly reasonable and
> beneficial.
>
> On Thu, Aug 31, 2017 at 8:11 AM, Ashish Vijaywargiya
> <[hidden email]> wrote:
>> +1, Taher. I will wait for your comment on Jacques question, we already
>> have a document but I think the automated script that can be implemented
>> here. Liquidbase and flyway seem to be promising solutions!
>>
>> One question always comes to my mind: Can we say that automated scripts
>> will support the migration from last two or at max three known releases?
>> I think we should not put the effort in building the migration script that
>> could migrate ofbiz earlier version(Let's say Ofbiz 10 or 9 or 4) to the
>> latest version. Please share your thoughts on this.
>>
>> Kind Regards
>> Ashish Vijaywargiya
>> HotWax Systems - est. 1997 <http://www.hotwaxsystems.com/>
>>
>>
>> On Wed, Aug 30, 2017 at 7:54 PM, Taher Alkhateeb <[hidden email]
>>> wrote:
>>> Good idea! Why not take it a step further, and write data migration
>>> scripts? They will serve two purposes in one: 1) document changes 2)
>>> automate upgrades.
>>>
>>> You can experiment with solutions like liquibase or flyway
>>>
>>> On Aug 30, 2017 4:23 PM, "Aditya Sharma" <[hidden email]>
>>> wrote:
>>>
>>> Hello everyone,
>>>
>>> For one of my assignments, I need to find out entity changes that took
>>> place between an older release and the latest release.
>>>
>>> One of the solutions that came up was comparing the database using MySQL
>>> Workbench or some other utility. I found around 60+ new entity changes and
>>> a lot of minor field changes since last big book was published (OFBiz 9 I
>>> suppose).
>>> It's fascinating for me that around 8 years passed since then and data
>>> model still stands well (Kudos to Universal Data Model that we followed in
>>> OFBiz)
>>>
>>>
>>> Just a proposal, since we don't have so many frequent changes in the data
>>> model. It will be good to have a page or some other method defined to keep
>>> a track of such changes.
>>>
>>> I feel such information can be quite helpful when migrating from an older
>>> to some newer release.
>>>
>>> Please share your thoughts.
>>>
>>> Thanks and Regards,
>>>
>>> *Aditya Sharma* | Enterprise Software Engineer
>>> HotWax Systems <http://www.hotwaxsystems.com/>
>>> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>>>

[Aditya Sharma]

Hello all,

I have created a ticket here
<https://issues.apache.org/jira/browse/OFBIZ-9902>.

I will be adding information about data model changes between OFBiz.9 to
OFBiz.17.11 in the initial file.

As commented on the ticket,

The file will have the format:


Entity Changes:

Here we will have a bulleted list with entity names and the revision number
1. Added Entities
2. Removed Entities


Field changes:

Here we will show data in tabular form in following format:

entityname | field | action | isPK | revision


Any suggestions are warmly welcomed.

Thanks and Regards,

*Aditya Sharma* | Enterprise Software Engineer
HotWax Systems <http://www.hotwaxsystems.com/>
<https://www.linkedin.com/in/aditya-sharma-78291810a/>

On Thu, Oct 12, 2017 at 3:02 AM, Michael Brohl <[hidden email]>
wrote:

> +1 for tracking datamodel changes together with data migration scripts
>
> In our customer projects, we track every change in a simple text file in
> the source code repository. It contains description of the changes,
> references to issues or requirement documentation and SQL scripts for
> migrations.
>
> In OFBiz, we should at least provide SQL scripts for the embedded Derby
> database, maybe there will be contributions for other databases as well.
>
> Something like db-changelog.derby.txt, db-changelog.mysql.txt or similar.
>
> Cheers,
>
> Michael
>
> Am 31.08.17 um 12:32 schrieb Taher Alkhateeb:
>
> Hi Ashish,
>>
>> With respect to Jacques' question, I kind of already answered in that
>> it is not only documentation but also automation.
>>
>> Now with respect to which releases to incorporate, it really depends
>> on what the community decides. I would personally prefer to not go any
>> earlier than 13, or preferably just 16 to trunk, which means we design
>> this solution for the future, not necessarily the past. The powerful
>> thing in using something like liquibase is that not only do you change
>> the schema (the entity engine can do that partially) but you also
>> decide on how to migrate the existing data to the new schema. For
>> example, you might need to split a field to two, or merge two fields,
>> and so on and so forth.
>>
>> Anyway, this is only an idea if people are interested in it. The
>> original idea of just documenting is also perfectly reasonable and
>> beneficial.
>>
>> On Thu, Aug 31, 2017 at 8:11 AM, Ashish Vijaywargiya
>> <[hidden email]> wrote:
>>
>>> +1, Taher. I will wait for your comment on Jacques question, we already
>>> have a document but I think the automated script that can be implemented
>>> here. Liquidbase and flyway seem to be promising solutions!
>>>
>>> One question always comes to my mind: Can we say that automated scripts
>>> will support the migration from last two or at max three known releases?
>>> I think we should not put the effort in building the migration script
>>> that
>>> could migrate ofbiz earlier version(Let's say Ofbiz 10 or 9 or 4) to the
>>> latest version. Please share your thoughts on this.
>>>
>>> Kind Regards
>>> Ashish Vijaywargiya
>>> HotWax Systems - est. 1997 <http://www.hotwaxsystems.com/>
>>>
>>>
>>> On Wed, Aug 30, 2017 at 7:54 PM, Taher Alkhateeb <
>>> [hidden email]
>>>
>>>> wrote:
>>>> Good idea! Why not take it a step further, and write data migration
>>>> scripts? They will serve two purposes in one: 1) document changes 2)
>>>> automate upgrades.
>>>>
>>>> You can experiment with solutions like liquibase or flyway
>>>>
>>>> On Aug 30, 2017 4:23 PM, "Aditya Sharma" <[hidden email]
>>>> om>
>>>> wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> For one of my assignments, I need to find out entity changes that took
>>>> place between an older release and the latest release.
>>>>
>>>> One of the solutions that came up was comparing the database using MySQL
>>>> Workbench or some other utility. I found around 60+ new entity changes
>>>> and
>>>> a lot of minor field changes since last big book was published (OFBiz 9
>>>> I
>>>> suppose).
>>>> It's fascinating for me that around 8 years passed since then and data
>>>> model still stands well (Kudos to Universal Data Model that we followed
>>>> in
>>>> OFBiz)
>>>>
>>>>
>>>> Just a proposal, since we don't have so many frequent changes in the
>>>> data
>>>> model. It will be good to have a page or some other method defined to
>>>> keep
>>>> a track of such changes.
>>>>
>>>> I feel such information can be quite helpful when migrating from an
>>>> older
>>>> to some newer release.
>>>>
>>>> Please share your thoughts.
>>>>
>>>> Thanks and Regards,
>>>>
>>>> *Aditya Sharma* | Enterprise Software Engineer
>>>> HotWax Systems <http://www.hotwaxsystems.com/>
>>>> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>>>>
>>>>
>
>

[Aditya Sharma]

Thank you all for your valuable inputs.

We have added the markdown file to trunk with all the data model changes
from Release 9 to Release 17 at R1818765 and it will soon be part of the
next release. The page is also added as child page to wiki page Revisions
Requiring Data Migration - upgrade ofbiz
<https://cwiki.apache.org/confluence/x/LoBr> as Data Migration by releases
<https://cwiki.apache.org/confluence/x/07JzB>

Thanks Sonal Patwari for the efforts put into gathering information for the
Big Book.

Thanks Jacques Le Roux, Michael Brohl, Pranay Pandey, and Deepak Dixit for
providing personal attention to the whole effort.


Cheers,

*Aditya Sharma* | Enterprise Software Engineer
HotWax Commerce <http://www.hotwax.co/> by HotWax Systems
<http://www.hotwaxsystems.com/>

<https://www.linkedin.com/in/aditya-sharma-78291810a/>

On Sat, Oct 28, 2017 at 12:13 PM, Aditya Sharma <
[hidden email]> wrote:

> Hello all,
>
> I have created a ticket here
> <https://issues.apache.org/jira/browse/OFBIZ-9902>.
>
> I will be adding information about data model changes between OFBiz.9 to
> OFBiz.17.11 in the initial file.
>
> As commented on the ticket,
>
> The file will have the format:
>
>
> Entity Changes:
>
> Here we will have a bulleted list with entity names and the revision number
> 1. Added Entities
> 2. Removed Entities
>
>
> Field changes:
>
> Here we will show data in tabular form in following format:
>
> entityname | field | action | isPK | revision
>
>
> Any suggestions are warmly welcomed.
>
> Thanks and Regards,
>
> *Aditya Sharma* | Enterprise Software Engineer
> HotWax Systems <http://www.hotwaxsystems.com/>
> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>
> On Thu, Oct 12, 2017 at 3:02 AM, Michael Brohl <[hidden email]>
> wrote:
>
>> +1 for tracking datamodel changes together with data migration scripts
>>
>> In our customer projects, we track every change in a simple text file in
>> the source code repository. It contains description of the changes,
>> references to issues or requirement documentation and SQL scripts for
>> migrations.
>>
>> In OFBiz, we should at least provide SQL scripts for the embedded Derby
>> database, maybe there will be contributions for other databases as well.
>>
>> Something like db-changelog.derby.txt, db-changelog.mysql.txt or similar.
>>
>> Cheers,
>>
>> Michael
>>
>> Am 31.08.17 um 12:32 schrieb Taher Alkhateeb:
>>
>> Hi Ashish,
>>>
>>> With respect to Jacques' question, I kind of already answered in that
>>> it is not only documentation but also automation.
>>>
>>> Now with respect to which releases to incorporate, it really depends
>>> on what the community decides. I would personally prefer to not go any
>>> earlier than 13, or preferably just 16 to trunk, which means we design
>>> this solution for the future, not necessarily the past. The powerful
>>> thing in using something like liquibase is that not only do you change
>>> the schema (the entity engine can do that partially) but you also
>>> decide on how to migrate the existing data to the new schema. For
>>> example, you might need to split a field to two, or merge two fields,
>>> and so on and so forth.
>>>
>>> Anyway, this is only an idea if people are interested in it. The
>>> original idea of just documenting is also perfectly reasonable and
>>> beneficial.
>>>
>>> On Thu, Aug 31, 2017 at 8:11 AM, Ashish Vijaywargiya
>>> <[hidden email]> wrote:
>>>
>>>> +1, Taher. I will wait for your comment on Jacques question, we already
>>>> have a document but I think the automated script that can be implemented
>>>> here. Liquidbase and flyway seem to be promising solutions!
>>>>
>>>> One question always comes to my mind: Can we say that automated scripts
>>>> will support the migration from last two or at max three known releases?
>>>> I think we should not put the effort in building the migration script
>>>> that
>>>> could migrate ofbiz earlier version(Let's say Ofbiz 10 or 9 or 4) to the
>>>> latest version. Please share your thoughts on this.
>>>>
>>>> Kind Regards
>>>> Ashish Vijaywargiya
>>>> HotWax Systems - est. 1997 <http://www.hotwaxsystems.com/>
>>>>
>>>>
>>>> On Wed, Aug 30, 2017 at 7:54 PM, Taher Alkhateeb <
>>>> [hidden email]
>>>>
>>>>> wrote:
>>>>> Good idea! Why not take it a step further, and write data migration
>>>>> scripts? They will serve two purposes in one: 1) document changes 2)
>>>>> automate upgrades.
>>>>>
>>>>> You can experiment with solutions like liquibase or flyway
>>>>>
>>>>> On Aug 30, 2017 4:23 PM, "Aditya Sharma" <
>>>>> [hidden email]>
>>>>> wrote:
>>>>>
>>>>> Hello everyone,
>>>>>
>>>>> For one of my assignments, I need to find out entity changes that took
>>>>> place between an older release and the latest release.
>>>>>
>>>>> One of the solutions that came up was comparing the database using
>>>>> MySQL
>>>>> Workbench or some other utility. I found around 60+ new entity changes
>>>>> and
>>>>> a lot of minor field changes since last big book was published (OFBiz
>>>>> 9 I
>>>>> suppose).
>>>>> It's fascinating for me that around 8 years passed since then and data
>>>>> model still stands well (Kudos to Universal Data Model that we
>>>>> followed in
>>>>> OFBiz)
>>>>>
>>>>>
>>>>> Just a proposal, since we don't have so many frequent changes in the
>>>>> data
>>>>> model. It will be good to have a page or some other method defined to
>>>>> keep
>>>>> a track of such changes.
>>>>>
>>>>> I feel such information can be quite helpful when migrating from an
>>>>> older
>>>>> to some newer release.
>>>>>
>>>>> Please share your thoughts.
>>>>>
>>>>> Thanks and Regards,
>>>>>
>>>>> *Aditya Sharma* | Enterprise Software Engineer
>>>>> HotWax Systems <http://www.hotwaxsystems.com/>
>>>>> <https://www.linkedin.com/in/aditya-sharma-78291810a/>
>>>>>
>>>>>
>>
>>
>