Web tools

[BJ Freeman]

I search the Jira but asking here in case I missed something.
when a complete entity export is done.The dependent entities are not sorted so
they are import after the parent Id are. I am not sure at this point if it is
entity specific.
example
     <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"
lastUpdatedStamp="2006-06-18 02:50:03.697" lastUpdatedTxStamp="2006-06-18
02:50:03.587" createdStamp="2006-04-10 14:21:32.027" createdTxStamp="2006-04-10
14:21:32.027"/>

is before
     <PartyType partyTypeId="Suppliers" hasTable="Y"
lastUpdatedStamp="2006-06-18 02:50:03.737" lastUpdatedTxStamp="2006-06-18
02:50:03.587" createdStamp="2006-04-10 14:19:13.297" createdTxStamp="2006-04-10
14:19:13.297"/>

so when it imports, there is a error and and the import for the file is rolled back

[cjhowe]

Hey BJ!

Long time no hear from ya. Welcome back!

I just happened to be using the webtools earlier this week and I can
confirm that is the way they are sorted and that it would probably be
better to sort these by createdStamp before writing the export file
instead of what I believe is the normal indexing by primary key.  I was
going to create a JIRA issue as soon as I created a patch, but feel
free.  The write order could be more intelligent, but I think the sort
by createdStamp will handle many of the situations without putting too
much additional thought behind it.

--- BJ Freeman <[hidden email]> wrote:

> I search the Jira but asking here in case I missed something.
> when a complete entity export is done.The dependent entities are not
> sorted so
> they are import after the parent Id are. I am not sure at this point
> if it is
> entity specific.
> example
>      <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"
> lastUpdatedStamp="2006-06-18 02:50:03.697"
> lastUpdatedTxStamp="2006-06-18
> 02:50:03.587" createdStamp="2006-04-10 14:21:32.027"
> createdTxStamp="2006-04-10
> 14:21:32.027"/>
>
> is before
>      <PartyType partyTypeId="Suppliers" hasTable="Y"
> lastUpdatedStamp="2006-06-18 02:50:03.737"
> lastUpdatedTxStamp="2006-06-18
> 02:50:03.587" createdStamp="2006-04-10 14:19:13.297"
> createdTxStamp="2006-04-10
> 14:19:13.297"/>
>
> so when it imports, there is a error and and the import for the file
> is rolled back
>

[BJ Freeman]

Create stamp might work unless you have someone like me that copies and pasts an
old record to fill in a new one.
in that case the keep time stamps on the import page should be disabled.
I have not given much thought yet.
first will go thru code to see what is being done first.
and Thanks.
Hopefully all the bad stuff is over and I can concentrate on the projects.


Chris Howe wrote:

> Hey BJ!
>
> Long time no hear from ya. Welcome back!
>
> I just happened to be using the webtools earlier this week and I can
> confirm that is the way they are sorted and that it would probably be
> better to sort these by createdStamp before writing the export file
> instead of what I believe is the normal indexing by primary key.  I was
> going to create a JIRA issue as soon as I created a patch, but feel
> free.  The write order could be more intelligent, but I think the sort
> by createdStamp will handle many of the situations without putting too
> much additional thought behind it.
>
> --- BJ Freeman <[hidden email]> wrote:
>
>> I search the Jira but asking here in case I missed something.
>> when a complete entity export is done.The dependent entities are not
>> sorted so
>> they are import after the parent Id are. I am not sure at this point
>> if it is
>> entity specific.
>> example
>>      <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"
>> lastUpdatedStamp="2006-06-18 02:50:03.697"
>> lastUpdatedTxStamp="2006-06-18
>> 02:50:03.587" createdStamp="2006-04-10 14:21:32.027"
>> createdTxStamp="2006-04-10
>> 14:21:32.027"/>
>>
>> is before
>>      <PartyType partyTypeId="Suppliers" hasTable="Y"
>> lastUpdatedStamp="2006-06-18 02:50:03.737"
>> lastUpdatedTxStamp="2006-06-18
>> 02:50:03.587" createdStamp="2006-04-10 14:19:13.297"
>> createdTxStamp="2006-04-10
>> 14:19:13.297"/>
>>
>> so when it imports, there is a error and and the import for the file
>> is rolled back
>>
>
>
>
>

[BJ Freeman]

I am looking the the current SVN code for webservices.
if i remember write there was a re-write to use cursors.
Now I see that there is simple code to get a full entity.
   // some databases don't support cursors, or other problems may happen, so if
there is an error here log it and move on to get as much as possible

Which brings us back to exporting large databases.
Or am I reading the code wrong?
If I am not, can code be added that returns if the database being used doe
support cursors. The put the code for cursors back in.
Or maybe add some code that does memory allocation checking so that the system
does not bomb when a entity data is very large.

Now to my problem the call to export is
       values = delegator.findListIteratorByCondition(curEntityName, null, null,
null, me.getPkFieldNames(), null);

will have to read up this to see if I can make the sort by parentID, That is
universal enough.
Kinda of grey in this are yet.


BJ Freeman wrote:

> Create stamp might work unless you have someone like me that copies and
> pasts an old record to fill in a new one.
> in that case the keep time stamps on the import page should be disabled.
> I have not given much thought yet.
> first will go thru code to see what is being done first.
> and Thanks.
> Hopefully all the bad stuff is over and I can concentrate on the projects.
>
>
> Chris Howe wrote:
>> Hey BJ!
>>
>> Long time no hear from ya. Welcome back!
>>
>> I just happened to be using the webtools earlier this week and I can
>> confirm that is the way they are sorted and that it would probably be
>> better to sort these by createdStamp before writing the export file
>> instead of what I believe is the normal indexing by primary key.  I was
>> going to create a JIRA issue as soon as I created a patch, but feel
>> free.  The write order could be more intelligent, but I think the sort
>> by createdStamp will handle many of the situations without putting too
>> much additional thought behind it.
>>
>> --- BJ Freeman <[hidden email]> wrote:
>>
>>> I search the Jira but asking here in case I missed something.
>>> when a complete entity export is done.The dependent entities are not
>>> sorted so they are import after the parent Id are. I am not sure at
>>> this point
>>> if it is entity specific.
>>> example
>>>      <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"
>>> lastUpdatedStamp="2006-06-18 02:50:03.697"
>>> lastUpdatedTxStamp="2006-06-18 02:50:03.587" createdStamp="2006-04-10
>>> 14:21:32.027"
>>> createdTxStamp="2006-04-10 14:21:32.027"/>
>>>
>>> is before
>>>      <PartyType partyTypeId="Suppliers" hasTable="Y"
>>> lastUpdatedStamp="2006-06-18 02:50:03.737"
>>> lastUpdatedTxStamp="2006-06-18 02:50:03.587" createdStamp="2006-04-10
>>> 14:19:13.297"
>>> createdTxStamp="2006-04-10 14:19:13.297"/>
>>>
>>> so when it imports, there is a error and and the import for the file
>>> is rolled back
>>>
>>
>>
>>
>>

[David E Jones]

Sorting a graph with loops to avoid any constraint breaking is an NP-
hard problem. In other words, there is no perfect solution and there  
are cases where even with very complicated efforts there is no  
guarantee of a solution.

This is why we have the dummy-fk and other such tools. You might also  
try turning on the initially deferred option for foreign key  
definitions in your database so that fks are not checked until the  
first phase of the commit.

-David


On Feb 24, 2007, at 2:25 PM, BJ Freeman wrote:

> I search the Jira but asking here in case I missed something.
> when a complete entity export is done.The dependent entities are  
> not sorted so they are import after the parent Id are. I am not  
> sure at this point if it is entity specific.
> example
>     <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"  
> lastUpdatedStamp="2006-06-18 02:50:03.697"  
> lastUpdatedTxStamp="2006-06-18 02:50:03.587"  
> createdStamp="2006-04-10 14:21:32.027" createdTxStamp="2006-04-10  
> 14:21:32.027"/>
>
> is before
>     <PartyType partyTypeId="Suppliers" hasTable="Y"  
> lastUpdatedStamp="2006-06-18 02:50:03.737"  
> lastUpdatedTxStamp="2006-06-18 02:50:03.587"  
> createdStamp="2006-04-10 14:19:13.297" createdTxStamp="2006-04-10  
> 14:19:13.297"/>
>
> so when it imports, there is a error and and the import for the  
> file is rolled back

[BJ Freeman]

thanks David.
I am using the default DB for testing.
I did try the  dummy-fk option and it did not seem to work in this instance.
I used only the two records I included here for testing.
I was not looking to put more code for looping, more passing thru to the db for
processing.
Maybe adding a new parm for passthru. and have that in the parms for the DB.
and a more elegant error message that might inform the Admin that data was to
large for the memory and possibly increase the memory var on startup.


David E. Jones wrote:

>
> Sorting a graph with loops to avoid any constraint breaking is an
> NP-hard problem. In other words, there is no perfect solution and there
> are cases where even with very complicated efforts there is no guarantee
> of a solution.
>
> This is why we have the dummy-fk and other such tools. You might also
> try turning on the initially deferred option for foreign key definitions
> in your database so that fks are not checked until the first phase of
> the commit.
>
> -David
>
>
> On Feb 24, 2007, at 2:25 PM, BJ Freeman wrote:
>
>> I search the Jira but asking here in case I missed something.
>> when a complete entity export is done.The dependent entities are not
>> sorted so they are import after the parent Id are. I am not sure at
>> this point if it is entity specific.
>> example
>>     <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"
>> lastUpdatedStamp="2006-06-18 02:50:03.697"
>> lastUpdatedTxStamp="2006-06-18 02:50:03.587" createdStamp="2006-04-10
>> 14:21:32.027" createdTxStamp="2006-04-10 14:21:32.027"/>
>>
>> is before
>>     <PartyType partyTypeId="Suppliers" hasTable="Y"
>> lastUpdatedStamp="2006-06-18 02:50:03.737"
>> lastUpdatedTxStamp="2006-06-18 02:50:03.587" createdStamp="2006-04-10
>> 14:19:13.297" createdTxStamp="2006-04-10 14:19:13.297"/>
>>
>> so when it imports, there is a error and and the import for the file
>> is rolled back
>

[BJ Freeman]

Update.
in the recent svn the the Import Text  has no options for:
  mostly insert
  or
Create "Dummy" FKs
here are the steps.
using New Derby DB
do a -install --readers=seed.
then use the webtools to import the xml files from folder that were exported
previously.
about 32 of the 240 files get imported.
when I try the import file or import the text I get errors.
As I said in the orginal post, I can get them to import by manually
repositioning the files.
I have not tried to import them  the --readers=custom yet.

So I will make a jira and see if we can get this to work, as you mentioned it
should.



BJ Freeman wrote:

> thanks David.
> I am using the default DB for testing.
> I did try the  dummy-fk option and it did not seem to work in this
> instance.
> I used only the two records I included here for testing.
> I was not looking to put more code for looping, more passing thru to the
> db for processing.
> Maybe adding a new parm for passthru. and have that in the parms for the
> DB.
> and a more elegant error message that might inform the Admin that data
> was to large for the memory and possibly increase the memory var on
> startup.
>
>
> David E. Jones wrote:
>>
>> Sorting a graph with loops to avoid any constraint breaking is an
>> NP-hard problem. In other words, there is no perfect solution and
>> there are cases where even with very complicated efforts there is no
>> guarantee of a solution.
>>
>> This is why we have the dummy-fk and other such tools. You might also
>> try turning on the initially deferred option for foreign key
>> definitions in your database so that fks are not checked until the
>> first phase of the commit.
>>
>> -David
>>
>>
>> On Feb 24, 2007, at 2:25 PM, BJ Freeman wrote:
>>
>>> I search the Jira but asking here in case I missed something.
>>> when a complete entity export is done.The dependent entities are not
>>> sorted so they are import after the parent Id are. I am not sure at
>>> this point if it is entity specific.
>>> example
>>>     <PartyType partyTypeId="Fullfilment" parentTypeId="Suppliers"
>>> lastUpdatedStamp="2006-06-18 02:50:03.697"
>>> lastUpdatedTxStamp="2006-06-18 02:50:03.587" createdStamp="2006-04-10
>>> 14:21:32.027" createdTxStamp="2006-04-10 14:21:32.027"/>
>>>
>>> is before
>>>     <PartyType partyTypeId="Suppliers" hasTable="Y"
>>> lastUpdatedStamp="2006-06-18 02:50:03.737"
>>> lastUpdatedTxStamp="2006-06-18 02:50:03.587" createdStamp="2006-04-10
>>> 14:19:13.297" createdTxStamp="2006-04-10 14:19:13.297"/>
>>>
>>> so when it imports, there is a error and and the import for the file
>>> is rolled back
>>