Semantics and need for local system datetime

[ariebovenberg]

The oddest of the DateTime classes is undoubtebly LocalDateTime. This is because its value is tied to the system timezone, which may change at any time.

This leads to tricky situations:

• a LocalDateTime can become non-existent after initialization
• a LocalDateTime can suddenly require disambiguation after initialization
• the meaning of the disambiguation can change (i.e. later is chosen, but the system timezone changes so that later is now a more/less further in time)
• converting from other aware types preserves the same moment in time, but can then abruptly change:

# always preserves the same moment in time, no matter now much you convert
original = UTCDateTime.now()
original.as_utc().as_offset().as_zoned("Asia/Toykyo").as_offset().as_zoned("Europe/Paris") == original

# however, as soon as you use `as_local()`, a race condition can occur in which the local datetime is shifted
intermediate = d.as_local()
os.environ['TZ'] = ... # set the system timezone to something different
intermediate.as_utc() == original  # false!

The current mititagations in place:

• a warning in the docs about the effect of changing the timezone
• all LocalDateTime methods check for existence first and raise exceptions.

Potential solutions:

• Put LocalDateTime in a separate category from the other aware types. Instead of as_local, make the method better reflect that you're entering a new reality.
• Name the class FloatingLocalDateTime to reflect how it 'floats' on the system timezone
• Have a way to (explicitly) automatically handle non-existent times
• an as_offset_local method to avoid race conditions when converting to local datetime at a particular moment.
• Make is necessary to call to_offset(dismabiguate=...) before comparison operations

[ariebovenberg]

One solution i'm gravitating towards: making LocalDateTime more like NaiveDateTime:

1. this is similar to the standard library, so it is recognizable for Python devs
2. a big case for using LocalDateTime is when the system timezone changes. It should be rock-solid reliable for this case.
3. disambiguation can't happen on instantiation, but explicitly every time it needs to be compared with 'aware' timezones to_offset(disambiguate=...) method. It's cumbersome, but the only correct way.
4. LocalDateTime.now() would actually make limited sense—as soon as it's determined, the moment in time could change. Perhaps it should take a different name...

Examples

Problem with the current situation

os.environ['TZ'] = 'America/New_York'
# this time exists just fine in NYC
backup_scheduled = LocalDateTime(2024, 3, 31, hour=2, minute=30)
os.environ['TZ'] = 'Europe/Paris'
# but it gets skipped in Paris DST transition!
should_run_backup = UTCDateTime.now() > backup_scheduled

The last line is unreliable, as the backup_scheduled could be non-existent depending on the system timezone. This is bad because:

• it's pretty unexpected that this exception is raised during comparison
• it doesn't allow the user to handle this more gracefully

What would the solved problem look like

os.environ['TZ'] = 'America/New_York'
backup_scheduled = LocalDateTime(2024, 3, 31, hour=2, minute=30)
os.environ['TZ'] = 'Europe/Paris'
# We are forced to call the `with_local_offset` method which offers various methods to deal with ambiguity/non-existence
should_run_backup = UTCDateTime.now() > backup_scheduled.with_local_offset(disambiguate=..., nonexistent=...)

[bxparks]

Side comment: Have you considered a different name for LocalDateTime? Because LocalDateTime in other popular libraries (Joda-Time, java.time, NodaTime) means a different thing. In those libraries, LocalDateTime is what is currently called NiaveDateTime in this library. There is too much potential for confusion.

Possible alternatives:

• SystemDateTime
• LocalizedDateTime

[ariebovenberg]

I'm coming around to this as well. Ideally I'd like consistent as_X and XDateTime.

• SystemDateTime/as_system()
• LocalSystemDateTime/as_local_system()
• LocalOSDateTime/as_local_os()
• OSLocalDateTime/as_os_local()

I'm less of a fan of Localized — the essential word here is "system"

[bxparks]

Following up the discussion in #59, and after reading the notes in this ticket, I'm still confused about the intended behavior of LocalDateTime/SystemDateTime when the OS timezone changes.

My expectation is that the timezone of the OS gets bound to the instance of the LocalDateTime/SystemDateTime at creation, and if the OS changes its timezone, it should not affect the instance of the LocalDateTime/SystemDateTime that I created earlier.

So I don't follow how the problems mentioned above can happen: "a LocalDateTime can become non-existent after initialization", and "a LocalDateTime can suddenly require disambiguation after initialization". My expectation is that the instance of LocalDateTime/SystemDateTime always remains valid.

[ariebovenberg]

My expectation is that the timezone of the OS gets bound to the instance of the LocalDateTime/SystemDateTime at creation, and if the OS changes its timezone, it should not affect the instance of the LocalDateTime/SystemDateTime that I created earlier.

I've come around to this as well, and in the 0.4 release candidate this has been changed. See the 0.4-dev docs here. Have a look at the API documentation of the LocalDateTime class.

[bxparks]

Sorry for being dense, but I'm still not understanding the intended behavior of LocalDateTime.

On the one hand, the document says: "Instances have the fixed offset of the system timezone at the time of initialization." That says to me that it's the same as an OffsetDateTime, with a fixed UTC offset that existed at the time of creation.

On the other hand, the doc also says: "Unlike OffsetDateTime, it knows about the system timezone and its DST transitions." That says to me that it's a ZonedDateTime with a tzinfo that defined the OS timezone at the time of creation.

I'm confused why the LocalDateTime/SystemDateTime cannot be just a ZonedDateTime, with its tzinfo set to be the OS tzinfo at the time of creation.

[ariebovenberg]

Sorry for being dense, but I'm still not understanding the intended behavior of SystemDateTime.

Not at all. SystemDateTime is obviously the most unusual of the classes. If it isn't clear to you, it probably isn't clear to the general public.

I'm confused why the SystemDateTime cannot be just a ZonedDateTime,

Assuming we keep ZonedDateTime for IANA timezones (which is a very common use case consistent across languages), this isn't possible. An operating system isn't in any way guaranteed to have a IANA-compatible timezone configured.

with its tzinfo set to be the OS tzinfo at the time of creation.

Note there there is no system tzinfo or ZoneInfo. There is only the ability to use datetime.astimezone(None) to convert to/from the local system timezone.

---

That says to me that it's the same as an OffsetDateTime, with a fixed UTC offset that existed at the time of creation.

The underlying data that SystemDateTime stores may be the same as OffsetDateTime, but its behavior is different. You can shift a SystemDateTime (i.e. it supports + and -) and it will account for the system DST. You cannot do this with OffsetDateTime.

Now, we could replace SystemDateTime with OffsetDateTime by adding methods like OffsetDateTime.now_local() and AwareDateTime.as_local_offset(). Downside is these methods give you an OffsetDateTime which doesn't support time shifting arithmetic.

Another niggle is that you'd have to add a special constructor to OffsetDateTime as well, and it would requires a disambiguate= argument. I'd like to keep ambiguity out of the OffsetDateTime object.

[bxparks]

I've been pondering why we have a disconnect on this topic, and I think it comes to this: I get the impression that you are restricting the ZonedDateTime class to support only IANA timezones (more precisely, it only supports subclasses of the tzinfo that handles only IANA timezones). Whereas, I tend to think of ZonedDateTime as capable of supporting any valid subclass of tzinfo. (This is probably the design philosophy that leads me to ask if ZonedDateTime should accept a tzinfo instead of a str, as wrote in #65).

From my understanding of Linux/Unix/POSIX machines, the system could be configured with an IANA timezone, but the system could also be configured with an arbitrary timezone rule such as the POSIX-style DST transition rules using the TZ environment variable. I am not familiar with how Python datetime supports such arbitrary timezone specifications. (To be honest, I have never used those datetime functions with tz=None. I have only used them with an explicit tzinfo from pytz, dateutil, or zoneinfo.) After reading the documentation for datetime.astimezone(tz=None) and the docs for the tzlocal package for example, it looks like Python datetime creates a special subclass of tzinfo to represent the "system timezone" on POSIX systems.

This is where I would say that the LocalDateTime/SystemDateTime should be just the ZonedDateTime class with whatever tzinfo class is needed to represent the "system" timezone. Because the behavioral difference between the "system timezone" and a valid "IANA timezone" is not in the DateTime class. The difference is fully defined by the tzinfo subclass.

It seems to me that primary reason that whenever needs two separate subclasses (LocalDateTime/SystemDateTime versus ZonedDateTime) is because the library has tightly coupled the concept of "time zones" (represented by tzinfo in Python) with the concept of the "date and time" system (represented by the Gregorian calendar and UTC time system DateTime). I think there is value in keeping these concepts as decoupled as possible. For one thing, it allows end-users to use a different timezone library (as I alluded to in #65), or even a completely different timezone classification that is not IANA.

[ariebovenberg]

I get the impression that you are restricting the ZonedDateTime class to support only IANA timezones

Yes. My approach is: "Different classes for different semantics":

• UTC and fixed offsets are unambiguous, linear through time, and integrate with ISO8601 an RFC3339.
• IANA zoned timezones are sometimes ambiguous. They are interoperable only thanks to tz IDs, which have near universal recognition.
• local time may also be ambiguous. It may be configured in various ways (IANA tz, POSIX string, or something else—you never know), and change at any time.
• highly customized datetimes. Advanced users may want to create arbitrary variable-offset zones. Because these are specific to the user, they are not interoperable with other systems.

Mixing semantics in one class is one of the problems with the standard library I'm aiming to avoid. The problem with a ZonedDateTime with arbitrary tzinfo is that it's semantics because overly broad. The limiting of functionality to clarify semantics is part of any well-designed library.

---

Back on the topic of SystemDateTime in particular: I'm on the fence whether to scrap the class or not. Here are my considerations at the moment:

The use cases

For Python programs, it is totally reasonable to want to interact with local system time. Here are the use cases I imagine:

1. converting to the local system timezone
2. converting from the local system timezone (including ambiguity)
3. performing arithmetic in local system timezone (i.e. when is 5 hours later)
4. convert a unix timestamp into the local system timezone
5. find out whether a time is ambiguous in the local system timezone

The alternatives

A: An immutable SystemDateTime class as currently implemented in 0.4-dev branch
B: Using OffsetDateTime to express system local datetimes

Comparison by use case

1. converting to the local system timezone:

# A
dt.as_local()
# B
dt.as_local_offset()

2. converting from the local system timezone (including ambiguity)

# A
LocalDateTime(2023, 10, 29, 2, 15, disambiguate="earlier")
# B
NaiveDateTime(2023, 10, 29, 2, 15).assume_local_offset(disambiguate="earlier")

3. performing arithmetic in local system timezone (i.e. when is 5 hours later)

# A
local_dt + hours(5)
# B
# note this isn't exactly the same for periods containing days, months, etc.
(local_dt.as_utc() + hours(5)).as_local_offset()

4. convert a unix timestamp into the local system timezone

# A
LocalDateTime.from_timestamp(ts)
# B
UTCDateTime.from_timestamp(ts).as_local_offset()

Other considerations

why drop the class:

• The semantics of LocalDateTime are not obvious at first glance
• There aren't many other libraries that have an equivalent. This is a signal that it may not be needed.

why keep the class:

• the class conveniently encapsulates interactions with the system timezone

[bxparks]

I get the impression that you are restricting the ZonedDateTime class to support only IANA timezones

Yes. My approach is: "Different classes for different semantics":

* UTC and fixed offsets are unambiguous, linear through time, and integrate with ISO8601 an RFC3339.

* IANA zoned timezones are sometimes ambiguous. They are interoperable only thanks to tz IDs, which have near universal recognition.

* local time may also be ambiguous. It may be configured in various ways (IANA tz, POSIX string, or something else—you never know), and change at any time.

* highly customized datetimes. Advanced users may want to create arbitrary variable-offset zones. Because these are specific to the user, they are not interoperable with other systems.

My argument is that those 4 things are exactly the same, in the sense that they are all DateTime that use the Gregorian calendar system and UTC time (more pedantically, POSIX time, because I don't think whenever deals with leap seconds). The semantic differences in those things are their timezones, or more accurately, how they choose to encode their UTC offsets (fixed, IANA, POSIX-style, or something custom).

[bxparks]

Let me present my thoughts from a different angle.

I think you have correctly identified the usability problem in the datetime.datetime library, which conflates "naive datetime" with the "aware datetime" into a single datetime class. Your NaiveDateTime and AwareDateTime separation is the right call.

Maybe I'm suggesting that the pendulum may have swung too far with the creation of numerous subclasses to model the different use cases of AwareDateTime. I tend to favor composition and aggregation instead of inheritance for most problems. I consider the various subclasses to be just an AwareDateTime with different TimeZone instances. Some of those subclasses are useful for ergonomic reasons for the programmer. To me, that's a different argument from saying that they capture different semantics.

[ariebovenberg]

My argument is that those 4 things are exactly the same, in the sense that they are all DateTime that use the Gregorian calendar system and UTC time (more pedantically, POSIX time, because I don't think whenever deals with leap seconds). The semantic differences in those things are their timezones, or more accurately, how they choose to encode their UTC offsets (fixed, IANA, POSIX-style, or something custom).

Absolutely agree. They are the same in that they represent a moment in POSIX timeline, and their "only" difference is how their UTC offset is defined.

I'd argue though, that it's exactly this difference that profoundly affects the behavior of the class! Addition/subtraction, parsing, formatting, and disambiguation are all but defined by their handling of the offset.

😉 Taking a cheeky straw-man metaphor here: imagine if a geometry library had one class Shape to describe squares, circles, triangles and the like. After all, they all have in common that they're bounded shapes in 2 dimensions. However, isn't it more valuable to tailor the classes to the differences instead of the commonalities? Having a radius attribute on Circle, but not on Square?

[ariebovenberg]

Maybe I'm suggesting that the pendulum may have swung too far with the creation of numerous subclasses to model the different use cases of AwareDateTime. I tend to favor composition and aggregation instead of inheritance for most problems. I consider the various subclasses to be just an AwareDateTime with different TimeZone instances.

I think we're getting to the core here. Would you consider that different TimeZone subclasses would be warranted?

Some of those subclasses are useful for ergonomic reasons for the programmer. To me, that's a different argument from saying that they capture different semantics.

Important not to confuse the two, I'll reflect on this.

[alejcas]

Hi, my bits:

I think this is extremely simple.

I think LocalDateTime should not exist:

• It is not always possible to know the local date time
• You can not pretend to have LocalDateTime to be dynamic. It should be up to the users to update this whenever is needed.
• whenever could implement some methods (there are plenty) to extract the system local datetime like tzlocal does. But this should return one of the other possible datetime classes (naive if no local datetime could be extracted, Iana or offset).
• but, at the end, the local date time is something that can be provided (or not) and it's up to the user to account for that.

Just my thoughts

Keep it simple

[alejcas]

Also I don't see a use case for UTCDateTime that can't be done with Zoned, Offset or Naive.

• Iana has a UTC timezone... so Zoned is in fact UTCDateTime
• Offset as well as it's +00:00
• and last, you can use Naive and interpret it as UTC.

I understand why you create a UTCDateTime class, but I don't think is necessary

For example:

utcdate = UTCDateTime(2022, 1, 1, 12)  # it's utc
offsetdate = OffsetDateTime(2022, 1, 1, 12, offset=hours(0))  # it's utc as well
zoneddate = ZonedDateTime(2022, 1, 1, 12, tz='UTC')  # it's utc as well

print(f'UTCDateTime is {utcdate}')
print(f'OffsetDateTime is {offsetdate}')
print(f'ZonedDateTime is {zoneddate}')

print(utcdate == offsetdate == zoneddate)

This outputs:

UTCDateTime is 2022-01-01T12:00:00Z
OffsetDateTime is 2022-01-01T12:00:00+00:00
ZonedDateTime is 2022-01-01T12:00:00+00:00[UTC]
True

So the difference here is:

1. The representation of the datetime ('Z' vs '+00:00' vs '+00:00[UTC]'):
   This is easy: a user should be able to get the representation he prefers from the other classes if possible.

2. The methods available in each class (from text representations, offset methods, to/from tz, etc.)
   This is also easy. I can't think of any method present in UTCDateTime that is not possible to implement in the other classes.

Either way, let the user decide if he wants UTC in Zoned, Offset or Naive datetimes...

Hope this adds up to the issue.

[ariebovenberg]

I think LocalDateTime should not exist:

• It is not always possible to know the local date time

Is this the case? Doesn't every OS have a system time? The python standard library (both time and datetime) don't mention any caveats. A system local time seems to be available at all times.

• You can not pretend to have LocalDateTime to be dynamic. It should be up to the users to update this whenever is needed.

Depends what exactly you mean with dynamic. I've already amended it to no longer 'automatically' adjust to changes in the system timezone. The user is in charge when to update this. See the API docs for upcoming release, what do you think?

• whenever could implement some methods (there are plenty) to extract the system local datetime like tzlocal does. But this should return one of the other possible datetime classes (naive if no local datetime could be extracted, Iana or offset).

The reason I chose for a fixed-offset is that this is what Python's astimezone(None) returns. Fixed offset seems like the most that can be relied on from an OS in general. tzlocal can always be used to try to get the IANA name.

• but, at the end, the local date time is something that can be provided (or not) and it's up to the user to account for that.

True that explicitly providing the timezone can be useful. In calendar applications, for example, you can have the user explicitly set it.
However, for smaller things like CLI tools, you can totally imagine wanting to just print a timestamp in whatever timezone the user expects, right?

print(f"Your download is expected to be ready at {LocalDateTime.now() + hours(1)")

Keep it simple

I can definitely relate to this feeling. What would you think of removing LocalDateTime from the README example code, and describing it on a special 'advanced features' page (along with other niche features).

[alejcas]

Well then LocalDateTime should disappear and just have something like the following.

There’s no need for a class for this.

from whenever import get_local_time

local_time = get_local_time()

print(type(local_time))
print(local_time)

Output:

OffsetDateTime
2022-01-01T12:00:00+00:00

[ariebovenberg]

Also I don't see a use case for UTCDateTime that can't be done with Zoned, Offset or Naive.

Correct, but: the value of a class isn't just in what it supports, but also in what it doesn't.

Example: imagine encountering this in a codebase:

class ChatMessage:
    sent: UTCDateTime
    text: str

This tells us that there is no local time stored with the message. This knowledge is invaluable for all sorts of features you might want to build: how to display such a time, how to store it in a database, etc.

[ariebovenberg]

Well then LocalDateTime should disappear and just have something like the following.

This is definitely something I've considered. Have a look at the second part of this comment: #30 (comment).

The reason LocalDateTime may be useful it that it makes certain actions clearer:

(legend: A = using LocalDateTime class, B = using OffsetDateTime instead)

1. converting to the local system timezone:

# A
dt.as_local()
# B
dt.as_local_offset()

2. converting from the local system timezone (including ambiguity)

# A
LocalDateTime(2023, 10, 29, 2, 15, disambiguate="earlier")
# B
NaiveDateTime(2023, 10, 29, 2, 15).assume_local_offset(disambiguate="earlier")

3. performing arithmetic in local system timezone (i.e. when is 5 hours later)

# A
local_dt + hours(5)
# B
# note this isn't exactly the same for periods containing days, months, etc.
(local_dt.as_utc() + hours(5)).as_local_offset()

4. convert a unix timestamp into the local system timezone

# A
LocalDateTime.from_timestamp(ts)
# B
UTCDateTime.from_timestamp(ts).as_local_offset()

[alejcas]

Also I don't see a use case for UTCDateTime that can't be done with Zoned, Offset or Naive.

Correct, but: the value of a class isn't just in what it supports, but also in what it doesn't.

Example: imagine encountering this in a codebase:

class ChatMessage:
    sent: UTCDateTime
    text: str

This tells us that there is no local time stored with the message. This knowledge is invaluable for all sorts of features you might want to build: how to display such a time, how to store it in a database, etc.

Well my system can definitely be set at utc but anyway, if you look it this way we should have 1000 classes to describe exactly what contents you should expect at a glance.

I think there’s a balance that for me goes into “practicality beats purity” 99% of the time.

A docstring will suffice for this usecase.

The other 3 classes (zoned, offset and naive) are really useful and I think everything fits inside.

Anyway, just my thoughts. Keep the great work!

[alejcas]

Well then LocalDateTime should disappear and just have something like the following.

This is definitely something I've considered. Have a look at the second part of this comment: #30 (comment).

The reason LocalDateTime may be useful it that it makes certain actions clearer:

(legend: A = using LocalDateTime class, B = using OffsetDateTime instead)

1. converting to the local system timezone:

# A
dt.as_local()
# B
dt.as_local_offset()

2. converting from the local system timezone (including ambiguity)

# A
LocalDateTime(2023, 10, 29, 2, 15, disambiguate="earlier")
# B
NaiveDateTime(2023, 10, 29, 2, 15).assume_local_offset(disambiguate="earlier")

3. performing arithmetic in local system timezone (i.e. when is 5 hours later)

# A
local_dt + hours(5)
# B
# note this isn't exactly the same for periods containing days, months, etc.
(local_dt.as_utc() + hours(5)).as_local_offset()

4. convert a unix timestamp into the local system timezone

# A
LocalDateTime.from_timestamp(ts)
# B
UTCDateTime.from_timestamp(ts).as_local_offset()

I work with local times all the time in my project O365 (using tzlocal extensively) and I really don’t see what a specific LocalDateTime class will help me with. I kind of don’t needed. In O365 I just store the local timezone and work with it. It’s a regular datetime…

I still think you don’t need a specific class for this. Again this is maybe a practicality vs purity kind of thing.

[alejcas]

IANA zoned timezones are sometimes ambiguous.

I don’t see how a datetime with IANA timezone could be ambiguous. Can you provide an example?

I think is the least ambiguous one

thanks

[ariebovenberg]

I don’t see how a datetime with IANA timezone could be ambiguous. Can you provide an example?

I could have been more clear. What I meant: constructing a IANA zoned datetime sometimes requires disambiguation. For example, the below datetime is ambiguous:

# at 3am the clock is moved 1 hour back (DST). 2:15am occurs twice
ZonedDateTime(2023, 10, 29, 2, 15, tz="Europe/Amsterdam")

Necessary to do:

ZonedDateTime(2023, 10, 29, 2, 15, tz="Europe/Amsterdam", disambiguate="later")

[alejcas]

I don’t see how a datetime with IANA timezone could be ambiguous. Can you provide an example?

Constructing a IANA zoned datetime sometimes requires disambiguation. For example, the below datetime is ambiguous:

# at 3am the clock is moved 1 hour back (DST). 2:15am occurs twice
ZonedDateTime(2023, 10, 29, 2, 15, tz="Europe/Amsterdam")

Necessary to do:

ZonedDateTime(2023, 10, 29, 2, 15, tz="Europe/Amsterdam", disambiguate="later")

Ok, True. I get it. Does any timezone format (iso, or whatever) encodes if it’s the earlier 2:15 or the later one?

[ariebovenberg]

Ok, True. I get it. Does any timezone format (iso, or whatever) encodes if it’s the earlier 2:15 or the later one?

AFAIK there are two approaches to encode this:

1. what python datetime does: a 'fold' parameter. fold=0 is the first, and fold=1 is the second
2. what everybody else seems to do: store the UTC offset.

You can see this in the canonical ZonedDateTime format: 2023-10-29 02:15:00+01:00[Europe/Amsterdam] vs. 2023-10-29 02:15:00+02:00[Europe/Amsterdam]

[alejcas]

Ok, I certainly prefer the offset.

The offset already encodes the fold + is something you can do math on (not with a fold flag).

In fact now that you mention it: ZonedDateTime should always expose the offset along with the timezone name.

[ariebovenberg]

ZonedDateTime should always expose the offset along with the timezone name.

Indeed, it does 😄

This reminds me of why UTCDateTime, OffsetDateTime, and ZonedDateTime all exist:

	UTCDateTime	OffsetDateTime	ZonedDateTime
represents a moment in time	X	X	X
represents a local time with UTC offset		X	X
Knows how UTC offset changes over time			X

[ariebovenberg]

An improved LocalSystemDateTime is now part of release 0.4