memcache write errors in the LMS ("object too large for cache")

[timmc-edx]

The vast majority of a certain class of memcache calls to set a key are failing with the error "object too large for cache".

These can be identified with @error.message:"b'object too large for cache'" "error.type:pymemcache.exceptions.MemcacheServerError in a Datadog query.

Notes

• The failing spans are all operation_name:memcached.command resource_name:set. These come from the memcache library integration. These failing writes do not propagate their error upwards, which is for the best but does mean that querying is a little complicated; to get more information about what memcache operation was attempted, you'll need to look at their parent spans, which are operation_name:django.cache. You'll need to do an a => b trace search.
• At the django.cache level I see that the resource names seem to all be django.core.cache.backends.memcached.OPERATION KEY_PREFIX (note the space). There are three key prefixes in effect: default, course_structure, and (uncommonly) general.
• The vast majority of these errors are coming from set on course_structure. Here's a status breakdown for those resources. A few of the errors come from default.
• Slicing a different way, almost all of course_structure sets are failing; almost all of default sets are succeeding. They are of roughly equal number.