Global Allsky map sanity checks?

[schoerg]

Hi,

not sure on where I should post this, some improvements for the global Allsky map: https://www.thomasjacquin.com/allsky-map/

• Option to hide cameras which do not provide a public image
• Hide cameras which didn't post an image in the last 24 hours (or some other range)

Not sure on how the last point can be adressed, right now the JSON looks like this:

    {
        "id": "41",
        "location": "Whitehorse YT",
        "owner": "Thomas Jacquin",
        "latitude": "60.59",
        "longitude": "-134.94",
        "website_url": "https:\/\/www.thomasjacquin.com\/allsky",
        "image_url": "https:\/\/www.thomasjacquin.com\/allsky\/image.jpg",
        "camera": "ZWO 224MC",
        "lens": "Arecont 1.55",
        "computer": "Raspberry Pi 3 Rev. 1.2"
    },

Which would require the client to check if the image exists and/or checking the HTTP Last modified timestamp.

[EricClaeys]

@schoerg, thanks for the suggestions.

The map currently only includes cameras that have updated their information in the last 30 days, i.e., the Pi has run postToMap.sh. When Allsky is running it runs postToMap.sh every other day. It wouldn't be practical to check the image update time of all images - it would likely take a minute or more.

@Alex-developer has a version of the map with a bunch of bells and whistles, including the ability to hide cameras without public images.
Alex, how close is that version to be shown to the public?

[schoerg]

Quick script to check all images, based on Last-modified header:

https://gist.github.com/schoerg/2bbd2f1631d5e1b1ab5b1b0b1434b5db it takes ~15s, but this would have to be done on the server.

Example output: https://gist.github.com/schoerg/f063dee1eb764277d7c6551e9e0def80

[EricClaeys]

Interesting data. Thanks for doing this. Allsky does some checks on the URL prior to uploading the data to the map, and I've wanted to add some checks to the server when it receives the data. Doing something like you did but in PHP (what the server uses) and then storing the timestamp would make it easy to filter based on it (your second request).

Can you determine how many URLs timed out?

[schoerg]

I have updated the script a bit, some stats:

432 total
278 with image URL

Errors:
45 - image older than 6 hours
20 - no HTTP 200
16 - connection Errors (timeout, no server at remote, not resolvable, wrong URL, 1 entry has allsky1.local as URL)
4 - no Last modified header

I don't know how much access you have on the server, my suggestion would be a cronjob for that. If you check at upload time (when allsky reports in) you'll get old data until the next report.
It only uses HTTP HEAD, so the image is not downloaded, which would probably allow to query every station every few hours.

Just personal opinion: I want to see how the sky looks like at the same time as my device, so even images older than 2 hours are not that useful.

[EricClaeys]

Made changes to display summary. Using your data.json:

num_ok = 191
num_no_url = 278
num_not_200 = 21
num_no_last_modified = 3
num_error = 19
num_too_old = 44 (> 6 old)

map.py.txt

If a Pi sends updated data, its image will almost always be current at that moment in time.
For what the map is for, data that is a few days or even a couple weeks old is fine. Many people don't run their Pi all the time or are changing things so the data may be old for "a while". That's why we have a cutoff of 30 days. 15 might be a better number, not sure.

For privacy reasons, I don't think the map should indicate the last update or last image without first requesting approval from the user. We're going to add a privacy statement to Allsky and ask for various permissions (like most pages do now with cookies), so including that data would only be from users who opt in.

[schoerg]

For clarity I would rename num_no_url to num_url, as in currently shows stations with an image_url, not without.

If it's only a display thing (meaning that Pis can get added again if they respond) I would go way lower then 15 days.

Good idea, although that data is there to begin with even now.

[Alex-developer]

This one? ;-)

I need to sort the Aurora display as its a real frame rate killer but other than that its good to go

Alex

[EricClaeys]

Yes. I think if we replace the current map with this one, only the "Show countries" option should be set.