dates in seed xml files are very broken

[Adrian Crum]

On 6/17/2010 8:37 AM, Adam Heath wrote:

> Adrian Crum wrote:
>> --- On Wed, 6/16/10, Adrian Crum<[hidden email]>  wrote:
>>> On 6/16/2010 3:53 PM, Adam Heath
>>> wrote:
>>>> Adrian Crum wrote:
>>>>> On 6/16/2010 3:40 PM, Adam Heath wrote:
>>>>>> Adrian Crum wrote:
>>>>>>> On 6/16/2010 3:24 PM, Adam Heath wrote:
>>>>>>>> Adrian Crum wrote:
>>>>>>>>> On 6/16/2010 3:03 PM, Adrian Crum
>>> wrote:
>>>>>>>>>> On 6/16/2010 2:56 PM, Adam
>>> Heath wrote:
>>>>>>>>>>> Adrian Crum wrote:
>>>>>>>>>>>> On 6/16/2010 2:03 PM,
>>> Adam Heath wrote:
>>>>>>>>>>>>> Adrian Crum
>>> wrote:
>>>>>>>>>>>>>> It sounds to
>>> me like the process is broken, not the data. Why not
>>>>>>>>>>>>>> specify a time
>>> zone along with a file name when
>>> importing/exporting?
>>>>>>>>>>>>> Because you may
>>> have multiple timezones in the data.
>>>>>>>>>>>>> Timezones are not
>>> actually part of the data. They are only
>>>>>>>>>>>>> utilized
>>>>>>>>>>>>> during display, by
>>> adjusting the raw time by some offset amount.
>>>>>>>>>>>> Time zones are used in
>>> parsing the XML date-time string into a
>>>>>>>>>>>> Timestamp, and in
>>> converting a Timestamp to XML.
>>>>>>>>>>> Timestamp to XML is
>>> correct; the timezone is in the output.
>>>>>>>>>>> XML to Timestamp is
>>> sometimes correct, but only if there is an
>>>>>>>>>>> actual
>>>>>>>>>>> timezone in the data.
>>>>>>>>>>>
>>>>>>>>>>> With no timezone, the
>>> actual parsed time(seconds since epoch +
>>>>>>>>>>> fractional) changes
>>> depending on the timezone the parsing took
>>>>>>>>>>> place in.
>>>>>>>>>>>
>>>>>>>>>>> Consider an import from
>>> xml, with no timezones, an xml dump, then a
>>>>>>>>>>> comparison of the 2 file
>>> sets. Ignoring any stamp date files, and
>>>>>>>>>>> fixing an ordering of
>>> rows, the files will not have the same values.
>>>>>>>>>>>>> I think the xml
>>> parser/dumper is broken. Timestamps should not be
>>>>>>>>>>>>> utilized there,
>>> instead any time is GMT/UTC(offset 0).
>>>>>>>>>>>> That would solve your
>>> problem, but now you've made life hard on
>>>>>>>>>>>> everyone
>>>>>>>>>>>> else. If I want to
>>> create my own seed data, I have to convert all
>>>>>>>>>>>> of my
>>>>>>>>>>>> date-time data to GMT
>>> before entering it.
>>>>>>>>>>> Just put a timezone on the
>>> value. I'm talking about values that
>>>>>>>>>>> currently have no timezone
>>> setting at all. If it has no timezone,
>>>>>>>>>>> the
>>>>>>>>>>> default is GMT, not
>>> whatever the local machine is set to.
>>>>>>>>>> That would work. I was
>>> picturing a time-zone attribute in the entity
>>>>>>>>>> element.
>>>>>>>>> Oops, entity-engine-xml element.
>>>>>>>> Both then?
>>>>>>>>
>>>>>>>> timezone-in-value wins.
>>>>>>>> then fallback on timezone in
>>> entity-engine-xml.
>>>>>>>> then GMT.
>>>>>>> Time zone in value first, then time zone
>>> attribute in entity-engine-xml
>>>>>>> element, then server's time zone (to
>>> preserve backward compatibility).
>>>>>>> If you want to reference your data to GMT,
>>> then you can do that in the
>>>>>>> entity-engine-xml attribute.
>>>>>>>
>>>>>>> It's flexible and it doesn't break any
>>> existing code.
>>>>>> After adding support for parsing all this
>>> data, I would still end up
>>>>>> changing the existing xml data files that
>>> ofbiz ships by default,
>>>>>> however.
>>>>> Why? I think everyone is used to seed/demo data
>>> being referenced to
>>>>> their own time zone.
>>>> Because it's a bug.  The data is not specified
>>> fully.  An import/dump
>>>> cycyle will not have the same values(the dumped file
>>> has timezone
>>>> information in it).  An import of seed, change
>>> timezone(or copy the
>>>> dataset to a different machine in a different
>>> timezone), and import of
>>>>     seed again(possibly after some minor
>>> seed upgrade) causes duplicate
>>>> entries.  Namely, PartyContactMech will have 2
>>> values, because the
>>>> hour was off.
>>>>
>>>> It's a bug, it should be fixed.
>>> 1. Import existing seed data. Date-time values are
>>> referenced to
>>> server's time zone.
>>> 2. Export data, specifying date-time data is referenced to
>>> GMT.
>>> 3. Import data on a server located in another time zone,
>>> specify
>>> date-time data is referenced to GMT.
>>>
>>> The date-time data should be the same in both servers.
>>>
>>> I'm not sure what you mean by fixing the seed data. If you
>>> mean
>>> specifying time zones in the seed data, then that will
>>> cause all kinds
>>> of problems. One simple example I can think of is the Staff
>>> Meeting
>>> recurring event in Work Effort. It is set for Monday at 10
>>> AM - that is
>>> clear in the seed data and in the documentation that
>>> references it. If a
>>> GMT time zone is specified in the seed data file, then any
>>> server
>>> outside of the GMT time zone will display the meeting at at
>>> the wrong time.
>>
>> Actually, this is a bad example - the Staff Meeting temporal expression does not contain a date-time value. A better example would be the recurring jobs that use a Frequency temporal expression.
>
> True, but lets expand on that.
>
> Let's say you have a time period definition, start work at 9am Monday
> morning, and stop work at 5pm Monday afternoon.  Let's say such a
> definition is entered into seed xml somewhere.  And lets then say that
> the 9am/5pm time frame is for an EST(United States Eastern).
>
> What happens when the server maintaining such a time period is hosted
> in California?

The California server loads seed data in its time zone and displays the
data in its time zone - 9AM to 5PM will be displayed as 9AM to 5PM. A
New York server loads seed data in its time zone and displays the data
in its time zone - 9AM to 5PM will be displayed as 9AM to 5PM. A New
Zealand server loads seed data in its time zone and displays the data in
its time zone - 9AM to 5PM will be displayed as 9AM to 5PM.

The seed data works fine. It has been working fine for years. It's the
process you're using that is the problem.

> If the above time period is encoded as timestamps, then the system is
> broken as designed.
>
> If, however, the time period is encoded as meta-data, then there is no
> problem.
>
> So, any timestamps that are used to encoded such extremely local
> specific time periods, needs to be fixed to *not* use timestamps.
>
> Another way of looking at the original problem, is that all time is
> the same.  5pm EST is the same as 4pm CST.  But with the way the
> current seed data is configured, the value will be different depending
> on a when/where you import it.

The Timestamp value will change depending on the time zone, but what is
displayed will remain the same. If the seed data is intended to contain
5PM, then 5PM is what you will get - no matter where you load it.

> The seed data also has problems when dealing with daylight savings
> time.  Some timezones have such a feature, and then there are local
> rules that specify when dst actually happens.  This was due to not
> forcing a timezone during import, and not forcing a timezone when testing.

Seed data for testing is another issue, and I agree that there is a
problem there.

-Adrian

[Adam Heath-2]

Adrian Crum wrote:

> On 6/17/2010 8:37 AM, Adam Heath wrote:
>> Adrian Crum wrote:
>>> --- On Wed, 6/16/10, Adrian Crum<[hidden email]>  wrote:
>>>> On 6/16/2010 3:53 PM, Adam Heath
>>>> wrote:
>>>>> Adrian Crum wrote:
>>>>>> On 6/16/2010 3:40 PM, Adam Heath wrote:
>>>>>>> Adrian Crum wrote:
>>>>>>>> On 6/16/2010 3:24 PM, Adam Heath wrote:
>>>>>>>>> Adrian Crum wrote:
>>>>>>>>>> On 6/16/2010 3:03 PM, Adrian Crum
>>>> wrote:
>>>>>>>>>>> On 6/16/2010 2:56 PM, Adam
>>>> Heath wrote:
>>>>>>>>>>>> Adrian Crum wrote:
>>>>>>>>>>>>> On 6/16/2010 2:03 PM,
>>>> Adam Heath wrote:
>>>>>>>>>>>>>> Adrian Crum
>>>> wrote:
>>>>>>>>>>>>>>> It sounds to
>>>> me like the process is broken, not the data. Why not
>>>>>>>>>>>>>>> specify a time
>>>> zone along with a file name when
>>>> importing/exporting?
>>>>>>>>>>>>>> Because you may
>>>> have multiple timezones in the data.
>>>>>>>>>>>>>> Timezones are not
>>>> actually part of the data. They are only
>>>>>>>>>>>>>> utilized
>>>>>>>>>>>>>> during display, by
>>>> adjusting the raw time by some offset amount.
>>>>>>>>>>>>> Time zones are used in
>>>> parsing the XML date-time string into a
>>>>>>>>>>>>> Timestamp, and in
>>>> converting a Timestamp to XML.
>>>>>>>>>>>> Timestamp to XML is
>>>> correct; the timezone is in the output.
>>>>>>>>>>>> XML to Timestamp is
>>>> sometimes correct, but only if there is an
>>>>>>>>>>>> actual
>>>>>>>>>>>> timezone in the data.
>>>>>>>>>>>>
>>>>>>>>>>>> With no timezone, the
>>>> actual parsed time(seconds since epoch +
>>>>>>>>>>>> fractional) changes
>>>> depending on the timezone the parsing took
>>>>>>>>>>>> place in.
>>>>>>>>>>>>
>>>>>>>>>>>> Consider an import from
>>>> xml, with no timezones, an xml dump, then a
>>>>>>>>>>>> comparison of the 2 file
>>>> sets. Ignoring any stamp date files, and
>>>>>>>>>>>> fixing an ordering of
>>>> rows, the files will not have the same values.
>>>>>>>>>>>>>> I think the xml
>>>> parser/dumper is broken. Timestamps should not be
>>>>>>>>>>>>>> utilized there,
>>>> instead any time is GMT/UTC(offset 0).
>>>>>>>>>>>>> That would solve your
>>>> problem, but now you've made life hard on
>>>>>>>>>>>>> everyone
>>>>>>>>>>>>> else. If I want to
>>>> create my own seed data, I have to convert all
>>>>>>>>>>>>> of my
>>>>>>>>>>>>> date-time data to GMT
>>>> before entering it.
>>>>>>>>>>>> Just put a timezone on the
>>>> value. I'm talking about values that
>>>>>>>>>>>> currently have no timezone
>>>> setting at all. If it has no timezone,
>>>>>>>>>>>> the
>>>>>>>>>>>> default is GMT, not
>>>> whatever the local machine is set to.
>>>>>>>>>>> That would work. I was
>>>> picturing a time-zone attribute in the entity
>>>>>>>>>>> element.
>>>>>>>>>> Oops, entity-engine-xml element.
>>>>>>>>> Both then?
>>>>>>>>>
>>>>>>>>> timezone-in-value wins.
>>>>>>>>> then fallback on timezone in
>>>> entity-engine-xml.
>>>>>>>>> then GMT.
>>>>>>>> Time zone in value first, then time zone
>>>> attribute in entity-engine-xml
>>>>>>>> element, then server's time zone (to
>>>> preserve backward compatibility).
>>>>>>>> If you want to reference your data to GMT,
>>>> then you can do that in the
>>>>>>>> entity-engine-xml attribute.
>>>>>>>>
>>>>>>>> It's flexible and it doesn't break any
>>>> existing code.
>>>>>>> After adding support for parsing all this
>>>> data, I would still end up
>>>>>>> changing the existing xml data files that
>>>> ofbiz ships by default,
>>>>>>> however.
>>>>>> Why? I think everyone is used to seed/demo data
>>>> being referenced to
>>>>>> their own time zone.
>>>>> Because it's a bug.  The data is not specified
>>>> fully.  An import/dump
>>>>> cycyle will not have the same values(the dumped file
>>>> has timezone
>>>>> information in it).  An import of seed, change
>>>> timezone(or copy the
>>>>> dataset to a different machine in a different
>>>> timezone), and import of
>>>>>     seed again(possibly after some minor
>>>> seed upgrade) causes duplicate
>>>>> entries.  Namely, PartyContactMech will have 2
>>>> values, because the
>>>>> hour was off.
>>>>>
>>>>> It's a bug, it should be fixed.
>>>> 1. Import existing seed data. Date-time values are
>>>> referenced to
>>>> server's time zone.
>>>> 2. Export data, specifying date-time data is referenced to
>>>> GMT.
>>>> 3. Import data on a server located in another time zone,
>>>> specify
>>>> date-time data is referenced to GMT.
>>>>
>>>> The date-time data should be the same in both servers.
>>>>
>>>> I'm not sure what you mean by fixing the seed data. If you
>>>> mean
>>>> specifying time zones in the seed data, then that will
>>>> cause all kinds
>>>> of problems. One simple example I can think of is the Staff
>>>> Meeting
>>>> recurring event in Work Effort. It is set for Monday at 10
>>>> AM - that is
>>>> clear in the seed data and in the documentation that
>>>> references it. If a
>>>> GMT time zone is specified in the seed data file, then any
>>>> server
>>>> outside of the GMT time zone will display the meeting at at
>>>> the wrong time.
>>>
>>> Actually, this is a bad example - the Staff Meeting temporal
>>> expression does not contain a date-time value. A better example would
>>> be the recurring jobs that use a Frequency temporal expression.
>>
>> True, but lets expand on that.
>>
>> Let's say you have a time period definition, start work at 9am Monday
>> morning, and stop work at 5pm Monday afternoon.  Let's say such a
>> definition is entered into seed xml somewhere.  And lets then say that
>> the 9am/5pm time frame is for an EST(United States Eastern).
>>
>> What happens when the server maintaining such a time period is hosted
>> in California?
>
> The California server loads seed data in its time zone and displays the
> data in its time zone - 9AM to 5PM will be displayed as 9AM to 5PM. A
> New York server loads seed data in its time zone and displays the data
> in its time zone - 9AM to 5PM will be displayed as 9AM to 5PM. A New
> Zealand server loads seed data in its time zone and displays the data in
> its time zone - 9AM to 5PM will be displayed as 9AM to 5PM.

The time period above is for EST, not PST.  I was trying to show that
it was a large company, with a presence in many places all over the
world.  To make everything work in such a situation, the initial data
load *must* contain timezones.

And if you are then s uggesting that the timestamps for such entities
in this situation must have timezones, but then other values don't
need them, then that way lies madness.

> The seed data works fine. It has been working fine for years. It's the
> process you're using that is the problem.

The process I am using is fine.  Ofbiz is growing, becoming bigger.
Just because no one has discovered this problem until now, does not
mean it doesn't exist, and shouldn't be fixed.  If we were to take
your way of thinking to it's conclusion, we would never be able to
implement new features.  I'm certain you don't mean that.

>> If the above time period is encoded as timestamps, then the system is
>> broken as designed.
>>
>> If, however, the time period is encoded as meta-data, then there is no
>> problem.
>>
>> So, any timestamps that are used to encoded such extremely local
>> specific time periods, needs to be fixed to *not* use timestamps.
>>
>> Another way of looking at the original problem, is that all time is
>> the same.  5pm EST is the same as 4pm CST.  But with the way the
>> current seed data is configured, the value will be different depending
>> on a when/where you import it.
>
> The Timestamp value will change depending on the time zone, but what is
> displayed will remain the same. If the seed data is intended to contain
> 5PM, then 5PM is what you will get - no matter where you load it.
>
>> The seed data also has problems when dealing with daylight savings
>> time.  Some timezones have such a feature, and then there are local
>> rules that specify when dst actually happens.  This was due to not
>> forcing a timezone during import, and not forcing a timezone when
>> testing.
>
> Seed data for testing is another issue, and I agree that there is a
> problem there.
>
> -Adrian

[Adam Heath-2]

> The seed data works fine. It has been working fine for years. It's the
> process you're using that is the problem.

Also, if my process is wrong, fine, that might be true.  But just
don't tell me it is wrong.  Give me a way to do what I have describe
countless times.

[Adrian Crum]

On 6/17/2010 9:09 AM, Adam Heath wrote:
>> The seed data works fine. It has been working fine for years. It's the
>> process you're using that is the problem.
>
> Also, if my process is wrong, fine, that might be true.  But just
> don't tell me it is wrong.  Give me a way to do what I have describe
> countless times.

I believe the reason for some of the confusion lies in the subject being
a moving target, or different subjects being blurred together. It might
help to make some clear distinctions:

1. Existing, out-of-the-box seed data works as expected. Date-time
values in the seed data are referenced to the server's time zone and
that data displays as intended or expected. Out-of-the-box seed data is
not broken - it has been working fine for years.

2. Entity data in XML format that contains date-time data will cause
problems if it is shared by servers in different time zones. The reason
is because no time zone is specified in the XML file, so each server
will reference date-time values to its own time zone. We need a way to
specify a time zone in the XML data so that each server will reference
the same time zone when importing the data.

3. Developers who wish to share entity data in XML format between
servers in different time zones will need to define a process that
specifies a time zone to be shared, and then ensure the shared XML data
contains the specified time zone. In other words, the specified time
zone is an anchor or key - all shared XML data uses that time zone.

To summarize what I've been saying:

1. You have identified a problem in sharing XML data that contains
date-time values and I agree it needs to be fixed. I suggest we specify
a time zone in the XML files, and if no time zone is specified, the
server's time zone is used.

2. The existing, out-of-the-box seed data files work fine. Please do not
specify time zones in the existing seed data files because that will
break a lot of existing code.

3. If you follow my suggestions and you're still having problems, then
there is something wrong in the process you're using.

I'm not trying to be argumentative or aloof - I'm really trying to help.
I just don't know how to make it any simpler.

-Adrian

[Adam Heath-2]

Adrian Crum wrote:

> On 6/17/2010 9:09 AM, Adam Heath wrote:
>>> The seed data works fine. It has been working fine for years. It's the
>>> process you're using that is the problem.
>>
>> Also, if my process is wrong, fine, that might be true.  But just
>> don't tell me it is wrong.  Give me a way to do what I have describe
>> countless times.
>
> I believe the reason for some of the confusion lies in the subject being
> a moving target, or different subjects being blurred together. It might
> help to make some clear distinctions:
>
> 1. Existing, out-of-the-box seed data works as expected. Date-time
> values in the seed data are referenced to the server's time zone and
> that data displays as intended or expected. Out-of-the-box seed data is
> not broken - it has been working fine for years.
>
> 2. Entity data in XML format that contains date-time data will cause
> problems if it is shared by servers in different time zones. The reason
> is because no time zone is specified in the XML file, so each server
> will reference date-time values to its own time zone. We need a way to
> specify a time zone in the XML data so that each server will reference
> the same time zone when importing the data.
>
> 3. Developers who wish to share entity data in XML format between
> servers in different time zones will need to define a process that
> specifies a time zone to be shared, and then ensure the shared XML data
> contains the specified time zone. In other words, the specified time
> zone is an anchor or key - all shared XML data uses that time zone.
>
> To summarize what I've been saying:
>
> 1. You have identified a problem in sharing XML data that contains
> date-time values and I agree it needs to be fixed. I suggest we specify
> a time zone in the XML files, and if no time zone is specified, the
> server's time zone is used.
>
> 2. The existing, out-of-the-box seed data files work fine. Please do not
> specify time zones in the existing seed data files because that will
> break a lot of existing code.

This is wrong.  Existing seed data *is* broken.

 user_login_id | group_id  |       from_date        | thru_date |
last_updated_stamp     |   last_updated_tx_stamp    |
created_stamp        |      created_tx_stamp
---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
 system        | FULLADMIN | 2001-01-01 13:00:00-05 |           |
2010-06-13 22:25:38.727-04 | 2010-06-13 22:25:38.664-04 | 2010-06-13
22:25:38.727-04 | 2010-06-13 22:25:38.664-04
 system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
20:30:20.83-04  | 2010-06-15 20:30:20.723-04
(3 rows)

This is output from psql, of UserLoginSecurityGroup.  Note the 1 hour
difference on the fromDate.  On 06-13, I imported all seed xml in
ofbiz on a machine in CDT.  I then did a pg_dump, copied the file to a
machine in EDT, imported it.  Then, did *just* a seed import on the
new machine.  No seed data was actually changed.  I now have duplicate
records.

> 3. If you follow my suggestions and you're still having problems, then
> there is something wrong in the process you're using.
>
> I'm not trying to be argumentative or aloof - I'm really trying to help.
> I just don't know how to make it any simpler.
>
> -Adrian

[Ean Schuessler]

BJ Freeman wrote:
> if you got that far it should be all four time fields.
> however I don't see the need even with sync to do this.
If you are talking about the entity sync fields then it definitely seems
like the lack of a timezone could cause your sync to fail.

--
Ean Schuessler, CTO
[hidden email]
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com

[Adrian Crum]

On 6/17/2010 9:42 AM, Adam Heath wrote:

> Adrian Crum wrote:
>> On 6/17/2010 9:09 AM, Adam Heath wrote:
>>>> The seed data works fine. It has been working fine for years. It's the
>>>> process you're using that is the problem.
>>>
>>> Also, if my process is wrong, fine, that might be true.  But just
>>> don't tell me it is wrong.  Give me a way to do what I have describe
>>> countless times.
>>
>> I believe the reason for some of the confusion lies in the subject being
>> a moving target, or different subjects being blurred together. It might
>> help to make some clear distinctions:
>>
>> 1. Existing, out-of-the-box seed data works as expected. Date-time
>> values in the seed data are referenced to the server's time zone and
>> that data displays as intended or expected. Out-of-the-box seed data is
>> not broken - it has been working fine for years.
>>
>> 2. Entity data in XML format that contains date-time data will cause
>> problems if it is shared by servers in different time zones. The reason
>> is because no time zone is specified in the XML file, so each server
>> will reference date-time values to its own time zone. We need a way to
>> specify a time zone in the XML data so that each server will reference
>> the same time zone when importing the data.
>>
>> 3. Developers who wish to share entity data in XML format between
>> servers in different time zones will need to define a process that
>> specifies a time zone to be shared, and then ensure the shared XML data
>> contains the specified time zone. In other words, the specified time
>> zone is an anchor or key - all shared XML data uses that time zone.
>>
>> To summarize what I've been saying:
>>
>> 1. You have identified a problem in sharing XML data that contains
>> date-time values and I agree it needs to be fixed. I suggest we specify
>> a time zone in the XML files, and if no time zone is specified, the
>> server's time zone is used.
>>
>> 2. The existing, out-of-the-box seed data files work fine. Please do not
>> specify time zones in the existing seed data files because that will
>> break a lot of existing code.
>
> This is wrong.  Existing seed data *is* broken.
>
>   user_login_id | group_id  |       from_date        | thru_date |
> last_updated_stamp     |   last_updated_tx_stamp    |
> created_stamp        |      created_tx_stamp
> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>   system        | FULLADMIN | 2001-01-01 13:00:00-05 |           |
> 2010-06-13 22:25:38.727-04 | 2010-06-13 22:25:38.664-04 | 2010-06-13
> 22:25:38.727-04 | 2010-06-13 22:25:38.664-04
>   system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
> (3 rows)
>
> This is output from psql, of UserLoginSecurityGroup.  Note the 1 hour
> difference on the fromDate.  On 06-13, I imported all seed xml in
> ofbiz on a machine in CDT.  I then did a pg_dump, copied the file to a
> machine in EDT, imported it.  Then, did *just* a seed import on the
> new machine.  No seed data was actually changed.  I now have duplicate
> records.

See #3 above.

>> 3. If you follow my suggestions and you're still having problems, then
>> there is something wrong in the process you're using.
>>
>> I'm not trying to be argumentative or aloof - I'm really trying to help.
>> I just don't know how to make it any simpler.
>>
>> -Adrian
>
>

[Adam Heath-2]

Ean Schuessler wrote:
> BJ Freeman wrote:
>> if you got that far it should be all four time fields.
>> however I don't see the need even with sync to do this.
> If you are talking about the entity sync fields then it definitely seems
> like the lack of a timezone could cause your sync to fail.

Not an issue.  I don't believe any xml data files have the stamp
fields specified.  They are always done with 'now', which doesn't have
a timezone, doesn't have a date, doesn't have a time.  It's just
seconds since epoch.  The date/time/tz that you see is just a figment
of your imagination, as the moment is the same no matter where you are.

Moving a database will keep the 'moment' correct, whatever the local
timezone is specified as.

[Ean Schuessler]

Adrian Crum wrote:
> Actually, this is a bad example - the Staff Meeting temporal expression does not contain a date-time value. A better example would be the recurring jobs that use a Frequency temporal expression.      
>  
The thing we need to focus on here is the entities that have a fromDate
as part of their primary key. If you examine these entries in the seed
data you can see that the current fromDates are largely arbitrary (ie.
Jan 1st, 2001) and are there only to fulfill the primary key.

I believe the point Adam is trying to make is that the moment in time
specified in seed data should not wiggle around depending on where the
import is run. The sets of primary key values specified in seed data
should not cause a duplication if the instance moves across timezones
and seed data is re-imported.

We should not confuse this issue (primary key value definitions in seed)
with specifying actual locally dependent time values (start of business
day, etc.). Those locally dependent time values obviously should reflect
their current timezone.

--
Ean Schuessler, CTO
[hidden email]
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com

[Ean Schuessler]

Adrian Crum wrote:
> The seed data works fine. It has been working fine for years. It's the
> process you're using that is the problem.
Just because something has been broken for a long time does not mean it
is fixed. We have found numerous long-standing problems that simply have
never been addressed. I'm sure you have found your share as well.

--
Ean Schuessler, CTO
[hidden email]
214-720-0700 x 315
Brainfood, Inc.
http://www.brainfood.com

[Adrian Crum]

On 6/17/2010 10:10 AM, Ean Schuessler wrote:
> Adrian Crum wrote:
>> The seed data works fine. It has been working fine for years. It's the
>> process you're using that is the problem.
> Just because something has been broken for a long time does not mean it
> is fixed. We have found numerous long-standing problems that simply have
> never been addressed. I'm sure you have found your share as well.

Someone please prove to me the existing seed data is broken. I can prove
that it isn't.

Using Adam's example...

In UserDemoData.xml:

<UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
fromDate="2001-01-01 12:00:00.0"/>

The Postgress dump from Adam's reply:

  user_login_id | group_id  |       from_date        | thru_date |
last_updated_stamp     |   last_updated_tx_stamp    |
created_stamp        |      created_tx_stamp
---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
  system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
20:30:20.83-04  | 2010-06-15 20:30:20.723-04

The fromDate was imported correctly. The seed data is not broken. What
is broken is the process used to transfer that data to a server in
another time zone.

-Adrian

[Adrian Crum]

On 6/17/2010 10:16 AM, Adrian Crum wrote:

> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>> Adrian Crum wrote:
>>> The seed data works fine. It has been working fine for years. It's the
>>> process you're using that is the problem.
>> Just because something has been broken for a long time does not mean it
>> is fixed. We have found numerous long-standing problems that simply have
>> never been addressed. I'm sure you have found your share as well.
>
> Someone please prove to me the existing seed data is broken. I can prove
> that it isn't.
>
> Using Adam's example...
>

Oops, grabbed the wrong seed data

In SecurityData.xml:

<UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="system"
fromDate="2001-01-01 12:00:00.0"/>

> In UserDemoData.xml:
>
> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
> fromDate="2001-01-01 12:00:00.0"/>
>
> The Postgress dump from Adam's reply:
>
> user_login_id | group_id | from_date | thru_date |
> last_updated_stamp | last_updated_tx_stamp |
> created_stamp | created_tx_stamp
> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>
> system | FULLADMIN | 2001-01-01 12:00:00-05 | |
> 2010-06-15 20:30:20.83-04 | 2010-06-15 20:30:20.723-04 | 2010-06-15
> 20:30:20.83-04 | 2010-06-15 20:30:20.723-04
>
> The fromDate was imported correctly. The seed data is not broken. What
> is broken is the process used to transfer that data to a server in
> another time zone.
>
> -Adrian
>

[Adam Heath-2]

Adrian Crum wrote:

> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>> Adrian Crum wrote:
>>> The seed data works fine. It has been working fine for years. It's the
>>> process you're using that is the problem.
>> Just because something has been broken for a long time does not mean it
>> is fixed. We have found numerous long-standing problems that simply have
>> never been addressed. I'm sure you have found your share as well.
>
> Someone please prove to me the existing seed data is broken. I can prove
> that it isn't.
>
> Using Adam's example...
>
> In UserDemoData.xml:
>
> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
> fromDate="2001-01-01 12:00:00.0"/>
>
> The Postgress dump from Adam's reply:
>
>  user_login_id | group_id  |       from_date        | thru_date |
> last_updated_stamp     |   last_updated_tx_stamp    |
> created_stamp        |      created_tx_stamp
> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>
>  system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
>
> The fromDate was imported correctly. The seed data is not broken. What
> is broken is the process used to transfer that data to a server in
> another time zone.

Your copy of my psql dump is incorrect.  There were 2 rows returned.
One was imported on 06-13(the initial import).  Then, on 06-15, the
seed xml data was imported a second time, but in a different tz.

[Adrian Crum]

On 6/17/2010 10:23 AM, Adam Heath wrote:

> Adrian Crum wrote:
>> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>>> Adrian Crum wrote:
>>>> The seed data works fine. It has been working fine for years. It's the
>>>> process you're using that is the problem.
>>> Just because something has been broken for a long time does not mean it
>>> is fixed. We have found numerous long-standing problems that simply have
>>> never been addressed. I'm sure you have found your share as well.
>>
>> Someone please prove to me the existing seed data is broken. I can prove
>> that it isn't.
>>
>> Using Adam's example...
>>
>> In UserDemoData.xml:
>>
>> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
>> fromDate="2001-01-01 12:00:00.0"/>
>>
>> The Postgress dump from Adam's reply:
>>
>>   user_login_id | group_id  |       from_date        | thru_date |
>> last_updated_stamp     |   last_updated_tx_stamp    |
>> created_stamp        |      created_tx_stamp
>> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>>
>>   system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
>> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
>> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
>>
>> The fromDate was imported correctly. The seed data is not broken. What
>> is broken is the process used to transfer that data to a server in
>> another time zone.
>
> Your copy of my psql dump is incorrect.  There were 2 rows returned.
> One was imported on 06-13(the initial import).  Then, on 06-15, the
> seed xml data was imported a second time, but in a different tz.

I can guarantee that is what your CDT server contains.

Anyways, I've done all I can do. I have a job I need to get back to. I
can reply again on Friday.

-Adrian

[Adam Heath-2]

Adrian Crum wrote:

> On 6/17/2010 10:23 AM, Adam Heath wrote:
>> Adrian Crum wrote:
>>> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>>>> Adrian Crum wrote:
>>>>> The seed data works fine. It has been working fine for years. It's the
>>>>> process you're using that is the problem.
>>>> Just because something has been broken for a long time does not mean it
>>>> is fixed. We have found numerous long-standing problems that simply
>>>> have
>>>> never been addressed. I'm sure you have found your share as well.
>>>
>>> Someone please prove to me the existing seed data is broken. I can prove
>>> that it isn't.
>>>
>>> Using Adam's example...
>>>
>>> In UserDemoData.xml:
>>>
>>> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
>>> fromDate="2001-01-01 12:00:00.0"/>
>>>
>>> The Postgress dump from Adam's reply:
>>>
>>>   user_login_id | group_id  |       from_date        | thru_date |
>>> last_updated_stamp     |   last_updated_tx_stamp    |
>>> created_stamp        |      created_tx_stamp
>>> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>>>
>>>
>>>   system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
>>> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
>>> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
>>>
>>> The fromDate was imported correctly. The seed data is not broken. What
>>> is broken is the process used to transfer that data to a server in
>>> another time zone.
>>
>> Your copy of my psql dump is incorrect.  There were 2 rows returned.
>> One was imported on 06-13(the initial import).  Then, on 06-15, the
>> seed xml data was imported a second time, but in a different tz.
>
> I can guarantee that is what your CDT server contains.
>
> Anyways, I've done all I can do. I have a job I need to get back to. I
> can reply again on Friday.

Sorta related, but StringToTimestamp converter is broken.  It doesn't
handle timezone parsing.  If the string ends with '.23', it'll change
that to end with '.023'.  However, if it ends with '.23-05', then the
fallback code will not change it to '.023-05'.


>
> -Adrian

[Adrian Crum]

On 6/17/2010 2:07 PM, Adam Heath wrote:

> Adrian Crum wrote:
>> On 6/17/2010 10:23 AM, Adam Heath wrote:
>>> Adrian Crum wrote:
>>>> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>>>>> Adrian Crum wrote:
>>>>>> The seed data works fine. It has been working fine for years. It's the
>>>>>> process you're using that is the problem.
>>>>> Just because something has been broken for a long time does not mean it
>>>>> is fixed. We have found numerous long-standing problems that simply
>>>>> have
>>>>> never been addressed. I'm sure you have found your share as well.
>>>>
>>>> Someone please prove to me the existing seed data is broken. I can prove
>>>> that it isn't.
>>>>
>>>> Using Adam's example...
>>>>
>>>> In UserDemoData.xml:
>>>>
>>>> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
>>>> fromDate="2001-01-01 12:00:00.0"/>
>>>>
>>>> The Postgress dump from Adam's reply:
>>>>
>>>>    user_login_id | group_id  |       from_date        | thru_date |
>>>> last_updated_stamp     |   last_updated_tx_stamp    |
>>>> created_stamp        |      created_tx_stamp
>>>> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>>>>
>>>>
>>>>    system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
>>>> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
>>>> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
>>>>
>>>> The fromDate was imported correctly. The seed data is not broken. What
>>>> is broken is the process used to transfer that data to a server in
>>>> another time zone.
>>>
>>> Your copy of my psql dump is incorrect.  There were 2 rows returned.
>>> One was imported on 06-13(the initial import).  Then, on 06-15, the
>>> seed xml data was imported a second time, but in a different tz.
>>
>> I can guarantee that is what your CDT server contains.
>>
>> Anyways, I've done all I can do. I have a job I need to get back to. I
>> can reply again on Friday.
>
> Sorta related, but StringToTimestamp converter is broken.  It doesn't
> handle timezone parsing.  If the string ends with '.23', it'll change
> that to end with '.023'.  However, if it ends with '.23-05', then the
> fallback code will not change it to '.023-05'.

What is timezone parsing? I agree it doesn't mimic the Timestamp.valueOf
method, but that is intentional to maintain backward compatibility.

[Adam Heath-2]

Adrian Crum wrote:

> On 6/17/2010 2:07 PM, Adam Heath wrote:
>> Adrian Crum wrote:
>>> On 6/17/2010 10:23 AM, Adam Heath wrote:
>>>> Adrian Crum wrote:
>>>>> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>>>>>> Adrian Crum wrote:
>>>>>>> The seed data works fine. It has been working fine for years.
>>>>>>> It's the
>>>>>>> process you're using that is the problem.
>>>>>> Just because something has been broken for a long time does not
>>>>>> mean it
>>>>>> is fixed. We have found numerous long-standing problems that simply
>>>>>> have
>>>>>> never been addressed. I'm sure you have found your share as well.
>>>>>
>>>>> Someone please prove to me the existing seed data is broken. I can
>>>>> prove
>>>>> that it isn't.
>>>>>
>>>>> Using Adam's example...
>>>>>
>>>>> In UserDemoData.xml:
>>>>>
>>>>> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
>>>>> fromDate="2001-01-01 12:00:00.0"/>
>>>>>
>>>>> The Postgress dump from Adam's reply:
>>>>>
>>>>>    user_login_id | group_id  |       from_date        | thru_date |
>>>>> last_updated_stamp     |   last_updated_tx_stamp    |
>>>>> created_stamp        |      created_tx_stamp
>>>>> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>>>>>
>>>>>
>>>>>
>>>>>    system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
>>>>> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
>>>>> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
>>>>>
>>>>> The fromDate was imported correctly. The seed data is not broken. What
>>>>> is broken is the process used to transfer that data to a server in
>>>>> another time zone.
>>>>
>>>> Your copy of my psql dump is incorrect.  There were 2 rows returned.
>>>> One was imported on 06-13(the initial import).  Then, on 06-15, the
>>>> seed xml data was imported a second time, but in a different tz.
>>>
>>> I can guarantee that is what your CDT server contains.
>>>
>>> Anyways, I've done all I can do. I have a job I need to get back to. I
>>> can reply again on Friday.
>>
>> Sorta related, but StringToTimestamp converter is broken.  It doesn't
>> handle timezone parsing.  If the string ends with '.23', it'll change
>> that to end with '.023'.  However, if it ends with '.23-05', then the
>> fallback code will not change it to '.023-05'.
>
> What is timezone parsing? I agree it doesn't mimic the Timestamp.valueOf
> method, but that is intentional to maintain backward compatibility.
>

I was going to say:

'2005-01-23 01:23:45.2-05' will be parsed incorrectly as '2005-01-23
01:23:45.002-05'.
'2005-01-23 01:23:45.2' will be parsed correctly as '2005-01-23
01:23:45.200'.

However, I'm noticing some oddities.  Issues with postgres 'timestamp'
vs 'timestamp with time zone', java.sql.Timestamp itself not having a
timezone whatsoever.  Need to spend more time.

[Adrian Crum]

On 6/17/2010 2:29 PM, Adam Heath wrote:

> Adrian Crum wrote:
>> On 6/17/2010 2:07 PM, Adam Heath wrote:
>>> Adrian Crum wrote:
>>>> On 6/17/2010 10:23 AM, Adam Heath wrote:
>>>>> Adrian Crum wrote:
>>>>>> On 6/17/2010 10:10 AM, Ean Schuessler wrote:
>>>>>>> Adrian Crum wrote:
>>>>>>>> The seed data works fine. It has been working fine for years.
>>>>>>>> It's the
>>>>>>>> process you're using that is the problem.
>>>>>>> Just because something has been broken for a long time does not
>>>>>>> mean it
>>>>>>> is fixed. We have found numerous long-standing problems that simply
>>>>>>> have
>>>>>>> never been addressed. I'm sure you have found your share as well.
>>>>>>
>>>>>> Someone please prove to me the existing seed data is broken. I can
>>>>>> prove
>>>>>> that it isn't.
>>>>>>
>>>>>> Using Adam's example...
>>>>>>
>>>>>> In UserDemoData.xml:
>>>>>>
>>>>>> <UserLoginSecurityGroup groupId="FULLADMIN" userLoginId="admin"
>>>>>> fromDate="2001-01-01 12:00:00.0"/>
>>>>>>
>>>>>> The Postgress dump from Adam's reply:
>>>>>>
>>>>>>     user_login_id | group_id  |       from_date        | thru_date |
>>>>>> last_updated_stamp     |   last_updated_tx_stamp    |
>>>>>> created_stamp        |      created_tx_stamp
>>>>>> ---------------+-----------+------------------------+-----------+----------------------------+----------------------------+----------------------------+----------------------------
>>>>>>
>>>>>>
>>>>>>
>>>>>>     system        | FULLADMIN | 2001-01-01 12:00:00-05 |           |
>>>>>> 2010-06-15 20:30:20.83-04  | 2010-06-15 20:30:20.723-04 | 2010-06-15
>>>>>> 20:30:20.83-04  | 2010-06-15 20:30:20.723-04
>>>>>>
>>>>>> The fromDate was imported correctly. The seed data is not broken. What
>>>>>> is broken is the process used to transfer that data to a server in
>>>>>> another time zone.
>>>>>
>>>>> Your copy of my psql dump is incorrect.  There were 2 rows returned.
>>>>> One was imported on 06-13(the initial import).  Then, on 06-15, the
>>>>> seed xml data was imported a second time, but in a different tz.
>>>>
>>>> I can guarantee that is what your CDT server contains.
>>>>
>>>> Anyways, I've done all I can do. I have a job I need to get back to. I
>>>> can reply again on Friday.
>>>
>>> Sorta related, but StringToTimestamp converter is broken.  It doesn't
>>> handle timezone parsing.  If the string ends with '.23', it'll change
>>> that to end with '.023'.  However, if it ends with '.23-05', then the
>>> fallback code will not change it to '.023-05'.
>>
>> What is timezone parsing? I agree it doesn't mimic the Timestamp.valueOf
>> method, but that is intentional to maintain backward compatibility.
>>
>
> I was going to say:
>
> '2005-01-23 01:23:45.2-05' will be parsed incorrectly as '2005-01-23
> 01:23:45.002-05'.
> '2005-01-23 01:23:45.2' will be parsed correctly as '2005-01-23
> 01:23:45.200'.
>
> However, I'm noticing some oddities.  Issues with postgres 'timestamp'
> vs 'timestamp with time zone', java.sql.Timestamp itself not having a
> timezone whatsoever.  Need to spend more time.

Another thing to watch out for is equating a date-time field type with a
Timestamp. There are no time zone rules defined for date-time fields,
while Timestamps are always referenced to GMT.