Encoding issues with many popular sites #434

pirate · 2019-01-23T10:29:35Z

Describe the bug

When using wayback in both --proxy and --proxy-record mode, many sites seem to display gibberish, possibly because of an incorrect encoding being forced somewhere.

Working sites:

Broken sites:

I'm not sure what makes some sites work and not others.

Steps to reproduce the bug

Use Python 3.7.2 and either pywb v2.1.0 or pywb-2.2.0.dev0 (latest develop branch)
Using an empty config.yaml, run:

wb-manager init demo
wayback --proxy-record --proxy demo

Open a page in a browser:

google-chrome --proxy-server=http://localhost:8080 --ignore-certificate-errors --disable-web-security https://cloudflare.com

Expected behavior

A readable site is recorded and replayed instead of ��

Screenshots

Environment

OS: macOS 10.14
Browser: Google Chrome 72.0.3626.64 beta
Version: Python 3.7.2 and pywb v2.1.0 [e.g. 22]

The text was updated successfully, but these errors were encountered:

N0taN3rd · 2019-01-24T21:14:44Z

Have you tried using a non-empty config file and ensured that pywb auto-generated a CA (docs) and or the dev branch?

I was not able to reproduce the issue using the config below when running from the dev branch using 3.7.2 and linux.

collections:
    all: $all
    live: $live

# Settings for each collection
use_js_obj_proxy: true

# Memento support, enable
enable_memento: true

# Replay content in an iframe
framed_replay: true

pirate · 2019-01-24T21:34:05Z

I just pulled and installed the dev branch, and it's still happening. I'm using 9597a63 aka pywb-2.2.0.dev0 now for this test.

It correctly autogenerated a CA, and the issue persists with the config you gave. I checked in both proxy and non-proxy mode and the issue is present:

And here are the logs while running it:

2019-01-24 16:39:06,229: [INFO]: Proxy recording into collection "squash"
2019-01-24 16:39:06,339: [INFO]: Starting Gevent Server on 8080
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Fcloudflare.com%2F&closest=now&matchType=exact HTTP/1.1" 200 1220 0.046708
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Fcloudflare.com%2F&closest=now&matchType=exact HTTP/1.1" 200 1242 0.064246
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Faccounts.google.com%2FListAccounts%3Fgpsia%3D1%26source%3DChromiumBrowser%26json%3Dstandard&closest=now&matchType=exact HTTP/1.1" 200 2166 0.070288
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Faccounts.google.com%2FListAccounts%3Fgpsia%3D1%26source%3DChromiumBrowser%26json%3Dstandard&closest=now&matchType=exact HTTP/1.1" 200 2181 0.086463
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Fclientservices.googleapis.com%2Fchrome-variations%2Fseed%3Fosname%3Dmac%26channel%3Dcanary%26milestone%3D73&closest=now&matchType=exact HTTP/1.1" 200 1417 0.106623
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Fclientservices.googleapis.com%2Fchrome-variations%2Fseed%3Fosname%3Dmac%26channel%3Dcanary%26milestone%3D73&closest=now&matchType=exact HTTP/1.1" 200 1439 0.124798
127.0.0.1 - - [2019-01-24 16:39:13] "CONNECT clientservices.googleapis.com:443 HTTP/1.1" 200 - 0.471312
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Fwww.cloudflare.com%2F&closest=now&matchType=exact HTTP/1.1" 200 17505 0.059444
127.0.0.1 - - [2019-01-24 16:39:13] "POST /live/resource/postreq?param.recorder.coll=squash&url=https%3A%2F%2Fwww.cloudflare.com%2F&closest=now&matchType=exact HTTP/1.1" 200 17527 0.068256

pirate · 2019-01-25T23:28:29Z

Here's an upload of my entire archive folder in case you want to see if it replays properly on a different machine:

wayback.zip

It's a fresh demo collection created with the config.yml you gave and the latest version of develop branch.

Thanks for helping debug this 👍

ikreymer · 2019-01-26T04:46:55Z

Thanks for sending the zip, unfortunately, can't repro the issue that you're seeing. The WARC in demo does contains valid HTML, but contains only the html for cloudflare, not any of the stylesheets (presumably since they didn't load in the browser).
I've tried both with latest Chrome and Chrome Canary, everything records and replays correctly.

Perhaps there is some other service running on your system that could interfere? It might be worth trying a different port than 8080, just in case. You can specify it with -p flag to change the port.
The HTML is Brotli encoded, but it should be getting decoded for rewriting, and pywb auto-decodes if the browser doesn't support Brotli anyway.

Other things to try: try upgrading the gevent library with pip install -U gevent in case there is an issue with that. The logs look a bit sparse, there should be more log entries made then that if everything is working.

pirate · 2019-01-26T04:53:40Z

Very strange, maybe it's a macOS thing? Or an encoding problem with my shell? What's your LC_ALL and LANG?

Is there a more verbose log mode I can use? I'll try upgrading gevent and running it in a fresh macOS user account and then a linux VM.

ikreymer · 2019-01-26T04:57:02Z

I'm on a mac as well.. LC_ALL is unset and LANG=en_US.UTF-8

ikreymer · 2019-01-26T05:00:26Z

This happens in url rewriting/non-proxy mode as well? What happens if you load http://localhost:8080/live/https://cloudflare.com/ either in the browser, or curl -L http://localhost:8080/live/mp_/https://cloudflare.com/ ?
Even without recording? and then with http://localhost:8080/demo/record/https://cloudflare.com/ ?

pirate · 2019-01-28T09:36:32Z

Two new findings: it only happens in Chrome, and I don't think it's an encoding issue. I tried decoding the page in a number of different encodings with Chrome, and none of them looked like real text. It's also not a font issue because copy pasting the output into a different program with a different font doesn't fix it. Could it be that it's not ungzipping the page for some reason in Chrome, and trying to show the compressed bytes as UTF-8 anyway?

While it appears to only happen in Chrome, it seems to happen reliably in all versions of Chrome (tested with fresh sessions & no extensions on Chrome Stable, Beta, and Canary).

It works fine in Safari & curl for both replaying and viewing, and in both proxy and rewriting modes. I would love to just use Safari, but unfortunately the primary reason I'm setting up pywb is to use it as a proxy recorder for scripted Chrome headless web archiving.

pirate · 2019-02-06T10:35:12Z

Update for anyone else following this issue, it turned out to be an issue with Brotli not being imported, so all pages using Brotli compression in chrome failed to decompress before playback/archiving, causing gibberish to show up.

Blocked by: python-hyper/brotlicffi#146

and also remove 'br' from any Accept-Encoding to avoid recording with brotli, addresses #434

* brotli: if the brotli module can not be loaded, print warning and also remove `br` from any Accept-Encoding header to avoid recording with brotli, addresses #434

ikreymer · 2019-02-28T19:41:38Z

The core issue was brotli not being installed properly. But, now pywb will not record in brotli if it can't decode it. Fixed in 2.2 release.

pirate · 2019-02-28T20:39:15Z

Thanks @ikreymer !

pirate mentioned this issue Jan 28, 2019

Extend WARC file with all requests made via all archive methods ArchiveBox/ArchiveBox#130

Open

ikreymer added a commit that referenced this issue Feb 19, 2019

brotli: if the brotli module can not be loaded, print warning

0ea5bea

and also remove 'br' from any Accept-Encoding to avoid recording with brotli, addresses #434

ikreymer mentioned this issue Feb 19, 2019

Brotli: Don't accept brotli if library can't be loaded. #444

Merged

8 tasks

ikreymer added this to the 2.2.0 milestone Feb 20, 2019

ikreymer closed this as completed Feb 28, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Encoding issues with many popular sites #434

Encoding issues with many popular sites #434

pirate commented Jan 23, 2019 •

edited

Loading

N0taN3rd commented Jan 24, 2019

pirate commented Jan 24, 2019 •

edited

Loading

pirate commented Jan 25, 2019

ikreymer commented Jan 26, 2019

pirate commented Jan 26, 2019

ikreymer commented Jan 26, 2019

ikreymer commented Jan 26, 2019 •

edited

Loading

pirate commented Jan 28, 2019 •

edited

Loading

pirate commented Feb 6, 2019 •

edited

Loading

ikreymer commented Feb 28, 2019

pirate commented Feb 28, 2019

Encoding issues with many popular sites #434

Encoding issues with many popular sites #434

Comments

pirate commented Jan 23, 2019 • edited Loading

Describe the bug

Steps to reproduce the bug

Expected behavior

Screenshots

Environment

N0taN3rd commented Jan 24, 2019

pirate commented Jan 24, 2019 • edited Loading

pirate commented Jan 25, 2019

ikreymer commented Jan 26, 2019

pirate commented Jan 26, 2019

ikreymer commented Jan 26, 2019

ikreymer commented Jan 26, 2019 • edited Loading

pirate commented Jan 28, 2019 • edited Loading

pirate commented Feb 6, 2019 • edited Loading

ikreymer commented Feb 28, 2019

pirate commented Feb 28, 2019

pirate commented Jan 23, 2019 •

edited

Loading

pirate commented Jan 24, 2019 •

edited

Loading

ikreymer commented Jan 26, 2019 •

edited

Loading

pirate commented Jan 28, 2019 •

edited

Loading

pirate commented Feb 6, 2019 •

edited

Loading