Now this is UX. Wonderful stuff.
And the site’s dark mode is fantastic…
Best dark mode ever!
Lol, who turned the lights out?
deleted by creator
This one really got a laugh out of me
Thank god for reader view because this makes me feel physically sick to look at.
I love it. People should be having more fun with their own personal sites.
deleted by creator
Is that other readers’ mouse pointers?
I love the comparison of string length of the same UTF-8 string in four programming languages (only the last one is correct, by the way):
Python 3:
len(“🤦🏼♂️”)
5
JavaScript / Java / C#:
“🤦🏼♂️”.length
7
Rust:
println!(“{}”, “🤦🏼♂️”.len());
17
Swift:
print(“🤦🏼♂️”.count)
1
That depends on your definition of correct lmao. Rust explicitly counts utf-8 scalar values, because that’s the length of the raw bytes contained in the string. There are many times where that value is more useful than the grapheme count.
And rust also has the “🤦”.chars().count() which returns 1.
I would rather argue that rust should not have a simple len function for strings, but since str is only a byte slice it works that way.
Also also the len function clearly states:
This length is in bytes, not chars or graphemes. In other words, it might not be what a human considers the length of the string.
None of these languages should have generic len() or size() for strings, come to think of it. It should always be something explicit like bytes() or chars() or graphemes(). But they’re there for legacy reasons.
That Rust function returns the number of codepoints, not the number of graphemes, which is rarely useful. You need to use a facepalm emoji with skin color modifiers to see the difference.
The way to get a proper grapheme count in Rust is e.g. via this library: https://crates.io/crates/unicode-segmentation
Makes sense, the code-points split is stable; meaning it’s fine to put in the standard library, the grapheme split changes every year so the volatility is probably better off in a crate.
Yeah, although having now seen two commenters with relatively high confidence claiming that counting codepoints ought be enough…
…and me almost having been the third such commenter, had I not decided to read the article first…
…I’m starting to feel more and more like the stdlib should force you through all kinds of hoops to get anything resembling a size of a string, so that you gladly search for a library.
Like, I’ve worked with decoding strings quite a bit in the past, I felt like I had an above average understanding of Unicode as a result. And I was still only vaguely aware of graphemes.
For what it’s worth, the documentation is very very clear on what these methods return. It explicitly redirects you to crates.io for splitting into grapheme clusters. It would be much better to have it in std, but I understand the argument that Std should only contain stable stuff.
As a systems programming language the .len() method should return the byte count IMO.
The problem is when you think you know stuff, but you don’t. I knew that counting bytes doesn’t work, but thought the number of codepoints was what I want. And then knowing that Rust uses UTF-8 internally, it’s logical that
.chars().count()gives the number of codepoints. No need to read documentation, if you’re so smart. 🙃It does give you the correct length in quite a lot of cases, too. Even the byte length looks correct for ASCII characters.
So, yeah, this would require a lot more consideration whether it’s worth it, but I’m mostly thinking there’d be no
.len()on the String type itself, and instead to get the byte count, you’d have to do.as_bytes().len().
Yeah, and as much as I understand the article saying there should be an easily accessible method for grapheme count, it’s also kind of mad to put something like this into a stdlib.
Its behaviour will break with each new Unicode standard. And you’d have to upgrade the whole stdlib to keep up-to-date with the newest Unicode standards.
It might make more sense to expose a standard library API for unicode data provided by (and updated with) the operating system. Something like the time zone database.
The way UTF-8 works is fixed though, isn’t it? A new Unicode standard should not change that, so as long as the string is UTF-8 encoded, you can determine the character count without needing to have the latest Unicode standard.Plus in Rust, you can instead use.chars().count()as Rust’s char type is UTF-8 Unicode encoded, thus strings are as well.turns out one should read the article before commenting
No offense, but did you read the article?
You should at least read the section “Wouldn’t UTF-32 be easier for everything?” and the following two sections for the context here.
So, everything you’ve said is correct, but it’s irrelevant for the grapheme count.
And you should pretty much never need to know the number of codepoints.yup, my bad. Frankly I thought grapheme meant something else, rather stupid of me. I think I understand the issue now and agree with you.
No worries, I almost commented here without reading the article, too, and did not really know what graphemes are beforehand either. 🫠
Nope, the article says that what is and is not a grapheme cluster changes between unicode versions each year :)
Unicode is thoroughly underrated.
UTF-8, doubly so. One of the amazing/clever things they did was to build off of ASCII as a subset by taking advantage of the extra bit to stay backwards compatible, which is a lesson we should all learn when evolving systems with users (your chances of success are much better if you extend than to rewrite).
On the other hand, having dealt with UTF-7 (a very “special” email encoding), it takes a certain kind of nerd to really appreciate the nuances of encodings.
I’ve recently come to appreciate the “refactor the code while you write it” and “keep possible future changes in mind” ideas more and more. I think it really increases the probability that the system can live on instead of becoming obsolete.
Yes, but once code becomes too spaghetti such that a “refactor while you write it” becomes too time intensive and error prone, it’s already too late.
Unrelated, but what do you think (if anything) might end up being used by the last remaining reserved bit in IP packet header flags?
https://en.wikipedia.org/wiki/Evil_bit
https://en.wikipedia.org/wiki/Internet_Protocol_version_4#Header
They believed 65,536 characters would be enough for all human languages.
Gotta love these kind of misjudgements. Obviously, they were pushing against pretty hard size restrictions back then, but at the same time, they did have the explicit goal of fitting in all languages and if you just look at the Asian languages, it should be pretty clear that it’s not a lot at all…
Man, Unicode is one of those things that is both brilliant and absolutely absurd. There is so much complexity to language and making one system to rule them all ends up involving so many compromises. Unicode has metadata for each character and algorithms dealing with normalization and capitalization and sorting. With human language being as varied as it is, these algorithms can have really wacky results. Another good article on it is https://eev.ee/blog/2015/09/12/dark-corners-of-unicode/
And if you want to RENDER text, oh boy. Look at this: https://faultlore.com/blah/text-hates-you/
Oh no, we’ve been hacked! Theres chinese character in the event log! Or was it just unicode?
The entire video is worth watching, the history of “Plain text” from the beginning of computing.
The mouse pointer background is kinda a dick move. Good article. but the background is annoying for tired old eyes - which I assume are a target demographic for that article.
Wow this is awful on mobile lol
js console:
document.querySelector('.pointers').hidden=true
Holy Jesus, what a color scheme.
I prefer it to black on white. Inferior to dark mode though.
Was actually a great read. I didn’t realize there were so many ways to encode the same character. TIL.
Because strings are such a huge problem nowadays, every single software developer needs to know the internals of them. I can’t even stress it enough, strings are such a burden nowadays that if you don’t know how to encode and decode one, you’re beyond fucked. It’ll make programming so difficult - no even worse, nigh impossible! Only those who know about unicode will be able to write any meaningful code.
currency symbols other than the $ (kind of tells you who invented computers, doesn’t it?)
Who wants to tell the author that not everything was invented in the US? (And computers certainly weren’t)
The stupid thing is, all the author had to do was write “kind of tells you who invented ASCII” and he’d have been 100% right in his logic and history.
Where were computers invented in your mind? You could define computer multiple ways but some of the early things we called computers were indeed invented in the US, at MIT in at least one case.
Well, it’s not really clear-cut, which is part of my point, but probably the 2 most significant people I could think of would be Babbage and Turing, both of whom were English. Definitely could make arguments about what is or isn’t considered a ‘computer’, to the point where it’s fuzzy, but regardless of how you look at it, ‘computers were invented in America’ is rather a stretch.
‘computers were invented in America’ is rather a stretch.
Which is why no one said that. I read most of the article and I’m still not sure what you were annoyed about. I didn’t see anything US-centric, or even anglocentric really.
To say I’m annoyed would be very much overstating it, just a (very minor) eye-roll at one small line in a generally very good article. Just the bit quoted:
currency symbols other than the $ (kind of tells you who invented computers, doesn’t it?)
So they could also be attributing it to some other country that uses
$for their currency, which is a few, but it seems most likely to be suggesting USD.
I think the author’s intended implication is absolutely that it’s a dollar because the USA invented the computer. The two problems I have is that:
- He’s talking about the American Standard Code for Information Interchange, not computers at that point
- Brits or Germans invented the computer (although I can’t deny that most of today’s commercial computers trace back to the US)
It’s just a lazy bit of thinking in an otherwise excellent and internationally-minded article and so it stuck out to me too.
If you go to the page without the trailing slash, the images don’t load
I do understant why old unicode versions re-used “i” and “I” for turkish lowercase dotted i and turkish uppercase dotless I, but I don’t understand why more recent version have not introduce two new characters that looks exactly the same but who don’t require locale-dependant knowlege to do something as basic as “to lowercase”.
Probably for the same reason Spanish used to consider ch, ll, and rr as a single character.
Just give me plain UTF32 with ~@4 billion code points, that really should be enough for any symbol ee can come up with. Give everything it’s own code point, no bullshit with combined glyphs that make text processing a nightmare. I need to be able to do a strlen either on byte length or amount of characters without the CPU spendings minute to count each individual character.
I think Unicode started as a great idea and the kind of blubbered into aimless “everybody kinda does what everyone wants” territory. Unicode is for humans, sure, but we shouldn’t forget that computers actually have to do the work
The article sure mentions 💩a lot.
I’m personally waiting for utf-64 and for unicode to go back to fixed encoding and forgetting about merging code points into complex characters. Just keep a zeptillion code points for absolutely everything.
