Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow HTTP/HTTPS URIs that do not end in .zip #6351

Open
ndoo opened this issue Dec 31, 2024 · 2 comments
Open

Allow HTTP/HTTPS URIs that do not end in .zip #6351

ndoo opened this issue Dec 31, 2024 · 2 comments

Comments

@ndoo
Copy link

ndoo commented Dec 31, 2024

Malaysia's GTFS feeds neither returns a ''Content-Type: application/zip'' header nor ends in a .zip extension (the API server performs 2 HTTP redirects which then contains the .zip extension).

URLs for testing can be found here: https://developer.data.gov.my/realtime-api/gtfs-static

Thus, the following line of code rejects GTFS API URLs for Malaysia if configured in ''build-config.json''.

if (httpsDataSourceMetadata.isZipContentType() || uri.getPath().endsWith(".zip")) {

There are some workarounds;

  1. Manually downloading the files with ''wget --trust-server-names'' before building
  2. Using the redirect destination URL https://openapi-malaysia-transport.s3.ap-southeast-1.amazonaws.com/mybas-johor/gtfs_mybas.zip which is not guaranteed to be stable

I have some suggestions to make this work:

  1. I have written to the Malaysian government to request addition of the Content-Type: application/zip header
  2. Add a flag to build-config.json -> transitFeeds -> ? e.g. ignoreFileExtension perhaps
  3. Perform URI validation later in the process, after any redirections have taken place
@vpaturet
Copy link
Contributor

vpaturet commented Jan 7, 2025

We have discussed this issue today during the developer meeting (which you are welcome to attend if you wish).
We do not plan to fix this any time soon due to limited development resources. We recommend that you either use one of the workarounds you suggested, or create a pull request to perform URI validation later in the process.

@ndoo
Copy link
Author

ndoo commented Jan 7, 2025

Thank you, I've written to the government provider of the GTFS feed to return the correct content-type and if I don't make any progress there I'll dive into the code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants