Repository navigation
Russian Webpage parsing support. #263
Description
- Platform: Mac
- Mercury Parser Version: Web based api (at moment)
- Node Version (if a Node bug):
- Browser Version (if a browser bug):
Expected Behavior
Proper encoding for Russian language.
Current Behavior
When parsing this link https://www.finam.ru/analysis/newsitem/putin-nagradil-grefa-ordenom-20190208-203615/?utm_source=rss&utm_medium=new_compaigns&utm_campaign=news_to_finamb it doesn't give proper encode output and hence format is messed up when rendering in html.
Steps to Reproduce
- Parse link https://www.finam.ru/analysis/newsitem/putin-nagradil-grefa-ordenom-20190208-203615/?utm_source=rss&utm_medium=new_compaigns&utm_campaign=news_to_finamb
- Check the content output
- Try to render that content with Cyrillic font
- You will see instead of proper format it shows bunch of '�'
Detailed Description
I use this API for parsing articles in my reader app. And there are some Russian news feed try to use and are not able to get proper format output.
Activity
mrgodhani commentedon Feb 8, 2019
More reference https://github.com/mrgodhani/raven-reader/issues/269
HenryQW commentedon Feb 11, 2019
The same applies to Chinese and other Asian languages, you get a bunch of unicodes rather than the actual content. See #264
adampash commentedon Feb 11, 2019
Thanks for reporting @mrgodhani — and @HenryQW. I'll be honest: I don't have a ton of experience with encoding in these scenarios. This is where the encoding currently takes place:
https://github.com/postlight/mercury-parser/blob/e033835c7287904371371f922c487e6d0d7d7db8/src/resource/index.js#L63-L79
Does anything stand out to you as doing it wrong? Or any other suggestions? We're more than happy to accept help.
vjyanand commentedon Feb 12, 2019
My findings #267
var ENCODING_RE = /charset=([\w-]+)\b/i;if (metaContentType && properEncoding !== encoding) {Thanks
mrgodhani commentedon Feb 12, 2019
@adampash I tested @vjyanand's fix and that's working well.
toufic-m commentedon Mar 4, 2019
This fix has been merged, and will be included in the next release.
mindfulme commentedon Aug 2, 2020
Did you get it released? I have the following when parsing Russian language articles
