[jira] Created: (OFBIZ-2866) Allow entity-data-reader to indicate if it should insert, update, or upsert records

[Nicolas Malin (Jira)]

Allow entity-data-reader to indicate if it should insert, update, or upsert records
-----------------------------------------------------------------------------------

                 Key: OFBIZ-2866
                 URL: https://issues.apache.org/jira/browse/OFBIZ-2866
             Project: OFBiz
          Issue Type: Improvement
          Components: framework
    Affects Versions: SVN trunk
            Reporter: Bob Morley


This ticket is the result of forum conversation: http://www.nabble.com/Handling-one-time-seed-data-tt25136333.html

What this support provides is the ability at the entity-data-reader level to indicate if the entities loaded from that reader should insert, update, or upsert records into the database.  The current process is an upsert.  This patch provides the following:

- enhancement on entity-data-reader to indicate if "insert-values" and "update-values" is true/false.  By default they are true/true; this provides the ability to re-execute "seed-initial" data to pickup new records (but leave others unmodified) by setting the attributes to true/false (effectively skipping updates).  In practice these are the only two flavors, but this design provides the flexibility to do all four permutations.

- resulting informational messages were showing the "read" total but calling this the "changed" total.  A new object was created (entityDataLoadResults) which can hold any metrics related to the data load.  Signature changes and some logic changes were put into place to use this result and then nicely report a true "written / read" information message during and at the end of the data load.

- a bug was found when the dirty check in DelegatorImpl.storeAll where it was using the GenericEntity.get method for the new/old value comparison.  This performed a resource lookup so these values would always match.  I found this when testing my code using the GeoData -- I was changing the country name and it was not being properly updated.  The fix was to change the protection on the GenericEntity.getFieldValue method and then use that in the DelegatorImpl.  This does a check of the true database value vs. the value form the data loading entity (typically xml file).

There should be no changes other than the information messages coming out in both the DataLoader and the WebToolsService.entityImportReaders.

Configuration change can be applied as such:
<entity-data-reader name="seed-initial" insert-values="true" update-values="false"/>

Sample of the information messages from a load:
00005 of 00029 from file:/Users/rmorley/dev/ofbiz-trunk/ofbiz/specialpurpose/assetmaint/data/AssetMaintDemoData.xml
=-=-=-=-=-=-= Finished the data load with 1086 rows changed of 12184 rows read.


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

     [ https://issues.apache.org/jira/browse/OFBIZ-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Bob Morley updated OFBIZ-2866:
------------------------------

    Attachment: OFBIZ-2866_EntityDataReaderInsertOrUpdate.patch

To try out this patch you must update your entityengine.xml so that one of your entity-data-reader elements has inert-values or update-values set to false.

> Allow entity-data-reader to indicate if it should insert, update, or upsert records
> -----------------------------------------------------------------------------------
>
>                 Key: OFBIZ-2866
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-2866
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: framework
>    Affects Versions: SVN trunk
>            Reporter: Bob Morley
>         Attachments: OFBIZ-2866_EntityDataReaderInsertOrUpdate.patch
>
>
> This ticket is the result of forum conversation: http://www.nabble.com/Handling-one-time-seed-data-tt25136333.html
> What this support provides is the ability at the entity-data-reader level to indicate if the entities loaded from that reader should insert, update, or upsert records into the database.  The current process is an upsert.  This patch provides the following:
> - enhancement on entity-data-reader to indicate if "insert-values" and "update-values" is true/false.  By default they are true/true; this provides the ability to re-execute "seed-initial" data to pickup new records (but leave others unmodified) by setting the attributes to true/false (effectively skipping updates).  In practice these are the only two flavors, but this design provides the flexibility to do all four permutations.
> - resulting informational messages were showing the "read" total but calling this the "changed" total.  A new object was created (entityDataLoadResults) which can hold any metrics related to the data load.  Signature changes and some logic changes were put into place to use this result and then nicely report a true "written / read" information message during and at the end of the data load.
> - a bug was found when the dirty check in DelegatorImpl.storeAll where it was using the GenericEntity.get method for the new/old value comparison.  This performed a resource lookup so these values would always match.  I found this when testing my code using the GeoData -- I was changing the country name and it was not being properly updated.  The fix was to change the protection on the GenericEntity.getFieldValue method and then use that in the DelegatorImpl.  This does a check of the true database value vs. the value form the data loading entity (typically xml file).
> There should be no changes other than the information messages coming out in both the DataLoader and the WebToolsService.entityImportReaders.
> Configuration change can be applied as such:
> <entity-data-reader name="seed-initial" insert-values="true" update-values="false"/>
> Sample of the information messages from a load:
> 00005 of 00029 from file:/Users/rmorley/dev/ofbiz-trunk/ofbiz/specialpurpose/assetmaint/data/AssetMaintDemoData.xml
> =-=-=-=-=-=-= Finished the data load with 1086 rows changed of 12184 rows read.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-2866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748114#action_12748114 ]

Bob Morley commented on OFBIZ-2866:
-----------------------------------

*NOTE* I forgot to put the Apache licensing to the new EntityDataLoadResults object.  Can you please add this if committing.

> Allow entity-data-reader to indicate if it should insert, update, or upsert records
> -----------------------------------------------------------------------------------
>
>                 Key: OFBIZ-2866
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-2866
>             Project: OFBiz
>          Issue Type: Improvement
>          Components: framework
>    Affects Versions: SVN trunk
>            Reporter: Bob Morley
>         Attachments: OFBIZ-2866_EntityDataReaderInsertOrUpdate.patch
>
>
> This ticket is the result of forum conversation: http://www.nabble.com/Handling-one-time-seed-data-tt25136333.html
> What this support provides is the ability at the entity-data-reader level to indicate if the entities loaded from that reader should insert, update, or upsert records into the database.  The current process is an upsert.  This patch provides the following:
> - enhancement on entity-data-reader to indicate if "insert-values" and "update-values" is true/false.  By default they are true/true; this provides the ability to re-execute "seed-initial" data to pickup new records (but leave others unmodified) by setting the attributes to true/false (effectively skipping updates).  In practice these are the only two flavors, but this design provides the flexibility to do all four permutations.
> - resulting informational messages were showing the "read" total but calling this the "changed" total.  A new object was created (entityDataLoadResults) which can hold any metrics related to the data load.  Signature changes and some logic changes were put into place to use this result and then nicely report a true "written / read" information message during and at the end of the data load.
> - a bug was found when the dirty check in DelegatorImpl.storeAll where it was using the GenericEntity.get method for the new/old value comparison.  This performed a resource lookup so these values would always match.  I found this when testing my code using the GeoData -- I was changing the country name and it was not being properly updated.  The fix was to change the protection on the GenericEntity.getFieldValue method and then use that in the DelegatorImpl.  This does a check of the true database value vs. the value form the data loading entity (typically xml file).
> There should be no changes other than the information messages coming out in both the DataLoader and the WebToolsService.entityImportReaders.
> Configuration change can be applied as such:
> <entity-data-reader name="seed-initial" insert-values="true" update-values="false"/>
> Sample of the information messages from a load:
> 00005 of 00029 from file:/Users/rmorley/dev/ofbiz-trunk/ofbiz/specialpurpose/assetmaint/data/AssetMaintDemoData.xml
> =-=-=-=-=-=-= Finished the data load with 1086 rows changed of 12184 rows read.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.