http://ofbiz.116.s1.nabble.com/importing-data-into-ofbiz-tp169514p169523.html
> I think what the entity engine tool does it the right way, and I'm not
> suggesting that we change it per se. The entity engine tool is very
> good for importing seed and demo data as we have it.
>
> There could be cases where a different process would be better. For
> example, if you're importing a lot of initial data (say 10,000 records),
> and not all of it is "clean" (say 10 bad records). The entity engine
> tool would basically fail on each bad record, so you'd go back, fix one
> record, try all 10,000 again, and repeat that ten times. It might be
> easier on the user if all 10,000 were read into a "temporary" table, the
> good rows imported, then the user fix the remaining bad rows in the
> database and insert those. The trouble with this "process" is that it
> becomes kind of custom and is harder to standardize. As I'm writing
> this I'm not sure what, if any of it, can be made part of the project.
>
> Si
>
> On Jul 10, 2006, at 10:27 AM, David E. Jones wrote:
>
>>
>> The point of transaction management is to have a set of operations
>> succeed or fail all together.
>>
>> Would it really be easier with a large or small file to have all of
>> the data go in except for the failures and then try to pick out all of
>> the failures and fix and get those in individually in another file or
>> by slimming down the original?
>>
>> Another problem with not failing the entire file is if there are
>> dependencies between the data or problems might be caused by having
>> partial data in the system, then you'll have cascading failures....
>>
>> So how would a partial failure be handled, or what sort of a process
>> would it fit into?
>>
>> -David
>>
>>
>> Si Chen wrote:
>>> Still, one issue I have is "exceptions handling": it seems that
>>> entity engine xml import would just rollback everything if one of the
>>> records to be imported is bad. Isn't better (or standard) to have a
>>> system where the data could be imported and then each exception
>>> handled individually?
>>> Si
>>> On Jul 7, 2006, at 1:27 PM, David E. Jones wrote:
>>>>
>>>> Yes, this looks like a good description of the general idea.
>>>>
>>>> -David
>>>>
>>>>
>>>> Si Chen wrote:
>>>>> David,
>>>>> Yes, the idea was to create something that would be better for some
>>>>> applications (namely, the one it's built for) than others.
>>>>> But I'm very curious about this entity-engine-transform-xml. So is
>>>>> this idea to paste a big block of XML into
>>>>> <entity-engine-transform-xml> file and then write the .FTL to do
>>>>> the transformation? And then run xml import on that file?
>>>>> And what about the freemarker template? Does it fall under their
>>>>> "Declarative XML Processing" documentation here:
>>>>>
http://freemarker.org/docs/xgui_declarative.html I looked at it
>>>>> briefly and recognized some of syntax.
>>>>> Si
>>>>> On Jul 7, 2006, at 12:03 PM, David E. Jones wrote:
>>>>>>
>>>>>> Stepping back a bit and looking at this conceptually:
>>>>>>
>>>>>> What you are proposing would be an import format that would work
>>>>>> well for a certain piece of software that the data is coming from.
>>>>>>
>>>>>> In fact, the normalized data model of OFBiz is far easier to
>>>>>> transform data to than a more normalized form would be.
>>>>>>
>>>>>> Consider Software A and Software B that both have very
>>>>>> functionality specific, redundant, and generally non-normalized
>>>>>> data models. The cardinality of fields relative to a product may
>>>>>> be different, and incompatible, and they may treat similar
>>>>>> information in very different ways. Because the OFBiz model is
>>>>>> more flexible it may be possible to migrate A->OFBiz and B->OFBiz
>>>>>> but it may not be possible to migrate A->B or B->A.
>>>>>>
>>>>>> So, yes, something is needed to transform incoming data but trying
>>>>>> to create something that is less normalized will just make it
>>>>>> easier for some systems (those that are close to it) and far more
>>>>>> difficult for other systems (that have more conceptual mismatches
>>>>>> with it).
>>>>>>
>>>>>> This is basically the unavoidable law of de-normalization.
>>>>>>
>>>>>> As far as tools and such go, FreeMarker is actually a pretty good
>>>>>> XML transformation tool. The Entity Engine import stuff does
>>>>>> already support using FTL files along with an incoming XML file to
>>>>>> pre-transform the data. There is an example in the
>>>>>> ofbiz/applications/ecommerce/data/DemoContent.xml file. Notice the
>>>>>> root element is "entity-engine-transform-xml" instead of
>>>>>> "entity-engine-xml".
>>>>>>
>>>>>> -David
>>>>>>
>>>>>>
>>>>>> Si Chen wrote:
>>>>>>> Hi everybody. I've been thinking about importing data into
>>>>>>> ofbiz. One of the issues is that we have a highly normalized
>>>>>>> data model which would require extensive mapping from external
>>>>>>> systems. Nevertheless, I think it is possible to create some
>>>>>>> standard "template" entities for importing data and thus create a
>>>>>>> standard practice and, eventually, standard tools for doing it.
>>>>>>> Here's what I'm thinking, given in the context of importing
>>>>>>> product data:
>>>>>>> 1. Create a standard non-normalized "holding" entity which
>>>>>>> contains productId, quantity on hand, available to promise,
>>>>>>> average cost, reorder point. Note that none of these are linked
>>>>>>> to any entity in ofbiz.
>>>>>>> 2. Add a timestamp fields to identify when the data was loaded
>>>>>>> into the "holding" entity and when it was imported into ofbiz.
>>>>>>> 3. Write a standard datafile xml import template which
>>>>>>> corresponds to this "holding" entity.
>>>>>>> 4. Write some processing script or service which takes new data
>>>>>>> from the holding entity and insert it into ofbiz. For example,
>>>>>>> in this case it would be to create InventoryItem,
>>>>>>> ProductAverageCost, and maybe some AcctgTrans (maybe . . .)
>>>>>>> It may be most efficient to write these holding entities
>>>>>>> corresponding to popular software programs where people might
>>>>>>> want to import data from, so they can mirror those programs' data
>>>>>>> layout more easily.
>>>>>>> Si
>
>