[jira] Created: (OFBIZ-757) entityImportDir does not retry failed files

[Nicolas Malin (Jira)]

entityImportDir does not retry failed files
-------------------------------------------

                 Key: OFBIZ-757
                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
             Project: OFBiz (The Open for Business Project)
          Issue Type: Bug
          Components: framework
            Reporter: Andrew Sykes
            Priority: Minor


It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

     [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Andrew Sykes updated OFBIZ-757:
-------------------------------

    Attachment: WebtoolsServices.java.patch

This patch fixes the problem, a count is kept of the number of failures per cycle, if all files fail then the import stops

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475389 ]

Chris Howe commented on OFBIZ-757:
----------------------------------

Thanks Andrew!  I was just frustrating myself with this issue yesterday.

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475408 ]

Andrew Sykes commented on OFBIZ-757:
------------------------------------

Very annoying isn't it?

Can you try it out for me Chris?

Thanks
- Andrew

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475502 ]

Chris Howe commented on OFBIZ-757:
----------------------------------

Hey Andrew,

It works well,  I especially like the better messaging.
I haven't had much look into the workings of those services yet.  I was wondering if it would be possible to do the same thing you did there but within the same file as well.

For instance when you have an entity that has a parentTypeId, the parent has to be loaded first otherwise referential integrity will fail and thus the rest of the file will fail (and thus all the files that rely on records in that file).  Do you have any insight in how to handle that?

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475582 ]

Andrew Sykes commented on OFBIZ-757:
------------------------------------

Chris,

To acheive your suggestion - which I admit would be a useful feature - would mean loading the file into memory and doing a RAM based shuffle of the records. Obviously this is risky if the file is big. More fundamentally this would mean moving from a SAX parser to a DOM parser, another problem is that at the moment a single file represents a single transaction, so if a line fails, the transaction is rolled back.

So, I guess the answer is, I'd love to see this feature, but I can't think of a way it could reasonably be acheived.

Perhaps a tool that split a mixed file into a set of single entity files could provide the answer?

On a related subject, I'm thinking It would be a good idea to add an option to read/write gzipped xml - have you any thoughts on this?

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12475591 ]

Chris Howe commented on OFBIZ-757:
----------------------------------

An option that would have helped initially is when you export data that it writes the records sorted by created date instead of alphanumerically by the first field in the definition.

is there a way to suspend the referential integrity check until after the file is loaded? then roll back if it fails instead of checking each line.  I'm thinking that would be more of a database specific thing.

I'm migrating from MySql to postrgres and it's proving quite difficult with the denormalized data being stuck onto the end of entities the way they are...(eg Party and UserLogin are referentially circular).  I think there is a cleaner way to do this that would also allow for integrity checking of the denormalized data.

I can't say I have any experience with gzipped xml.  Would you simply decompress it and read it like normal?

Direct migration from one delegator to another might be beneficial as well.  Because their is sync functionality, I would think this may be rather easy to accomplish.  

Anyway, just some more thoughts in this direction.  This patch works well double plus good :-)

vote +1

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

     [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jacques Le Roux updated OFBIZ-757:
----------------------------------

    Attachment: WebtoolsServices.java.patch

Andrew, Chris,

I reviewed, re-formatted (according to http://java.sun.com/docs/codeconv/html/CodeConvTOC.doc.html) and tested this patch. I believe it's a really important and nice improvement and I vote for.

BTW, testing I found the export option being ugly (too large fonts). I will report that on dev ML (this is of course not related to this pactch). I guess it's due to recent UI imporvement.

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch, WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

     [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Adrian Crum updated OFBIZ-757:
------------------------------

    Attachment: WebtoolsServices.java.patch

Same patch with UI fixed.

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch, WebtoolsServices.java.patch, WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

    [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476523 ]

Jacques Le Roux commented on OFBIZ-757:
---------------------------------------

Thanks Adrian, this solves the issue indeed.

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>            Priority: Minor
>         Attachments: WebtoolsServices.java.patch, WebtoolsServices.java.patch, WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Nicolas Malin (Jira)]

     [ https://issues.apache.org/jira/browse/OFBIZ-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

David E. Jones closed OFBIZ-757.
--------------------------------

       Resolution: Fixed
    Fix Version/s: SVN trunk
         Assignee: David E. Jones

Thanks Andrew, Jacques, Chris, and Adrian for creating and testing this. It appears that this has been tested by around 4 people, so in it goes ! (even though I haven't tested it...).

The latest patch is in SVN rev 513211.

> entityImportDir does not retry failed files
> -------------------------------------------
>
>                 Key: OFBIZ-757
>                 URL: https://issues.apache.org/jira/browse/OFBIZ-757
>             Project: OFBiz (The Open for Business Project)
>          Issue Type: Bug
>          Components: framework
>            Reporter: Andrew Sykes
>         Assigned To: David E. Jones
>            Priority: Minor
>             Fix For: SVN trunk
>
>         Attachments: WebtoolsServices.java.patch, WebtoolsServices.java.patch, WebtoolsServices.java.patch
>
>
> It looks like the code for entityImportDir should retry failed files in case referential integrity problems are fixed by a subsequent file, but this doesn't happen.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.