PiP in Maps: The Cultural Twist and Out-of-Box Thinking

I've always thought of Chinese apps as "more of an OS than an app," but recently realized that I judged based on mostly two apps: Wechat and Alipay. While it's probably unlikely that you see really simple apps like a funny-looking "random number generator" on a Chinese phone (they use Wechat mini-programs,) most everyday apps look similar enough to their Silicon Valley counterparts, with a slight twist to accommodate Chinese culture and lifestyle.

One example would be the Wechat Moment feature in which you can selectively show or hide your post to specific group of friends - it's now available on Facebook as well, but not back in 2015. It wouldn't cost a lot to implement yet would increase users' satisfaction a lot - regardless of cultural backgrounds, most people had gone through hard times when they wanted to vent within their 5 closest friends, wouldn't they? I'm still wondering why it took years for then-Facebook-now-Meta to keep up. Apart from the fact that Zuck wanted to spend the money on Metauniverse, it was possible that Americans are less likely than Asians to admit that they want this feature. The one feature on a Chinese app I used this year is the epitome of "localized UX design" that's "outside the box."

Welcome to Chinese map apps, which shows your route in as a floating window when you "background" the app to do something else... even before the existence of Dynamic Island.

I've used it in Tencent Maps, and according to my friends in China, it's available on Baidu and Gaode as well. How surprising (pun intended :P)

How It's Implemented

The first time I used it, I thought it looked similar to the "Picture in Picture" mode, but I looked into the developer docs and couldn't figure out how to do it without an actual video stream, and I thought it would be costly to "manually" convert some UIView to a video stream 😂 then I thought maybe it was the "Live Activities," but it looked different, and the timeline didn't match - according to the release notes, Tencent Map on iOS starts supporting Live Activities at the end of March, a while later than me seeing this "floating window" feature.

Then I realized that Google Meet on iOS looks like this...

Now that I know it's possible for PiP mode to show something that's probably not a video, a quick search on GitHub showed an example (UIPiPView) that did the work within less than 500 lines of code. The gist is that you turn an UIView to a "video stream" by the following transformations:

UIView -> CVPixelBuffer -> CMSampleBuffer -> enqueue to AVSampleBufferDisplayLayer

so indeed... it did the "costly" work of "converting UIView into a video stream" 😂 this is actually a bit harder than I have expected.

Why I think it's Brilliant

In my opinion, the best products are the ones you don't know you need until you try them. For those taking public transportation, that means reading your news or social media feed in peace, knowing that you won't miss your stop, especially on a bus instead of metro.

My experiences riding on bus in foreign countries had always included frequent switching between Google Maps and something else to kill time. I probably wouldn't have Google Maps open the entire ride because it would be boring, but I always needed to switch back and see which two stops I was between, and how many more stops to go. Having a "floating window" that keeps me updated over my newsfeed or something calms me down during the journey. It's proved to be something that I haven't known that I needed, and now I do.

I also think that building it through the PiP interface exemplifies "thinking outside the box," since that the interface probably isn't designed as a "all-purpose floating window" in the first place. "Geolocation update" and "video playing" sound so unrelated to each other that it takes a genius mind to connect the dots!

Why is it better than Live Activities in Dynamic Island?

1. "Live Activities" isn't designed to stay on the screen the entire time. (Techically, it can, so maybe this shouldn't count...)
2. Even if it does, you cannot move it around - it only sticks to the top, so unless you dismiss it, the top part of your screen would be hidden.

Let's talk about the Cons, of course.

I'm pretty sure compiling AVKit and having to convert views to videos real time takes a bunch of memory. Tallying with the most complained issue by my friends in China - their smartphones burn out really fast...

Culture and Localization

I'm concluding the article with this section because well, it contains a conclusion, and unfortunately, a disappointing one.

I started asking myself this question - why don't we have the "floating geolocation update" in American map apps?

As a mobile developer who works in the States, I'm trying to convince myself that it was for the following -

1. Wayyyy less people take public transportation, especially buses. And then on half of those old metros, your phone doesn't get stable signal so of course you don't get geolocation updates...
2. Performance over athetics. Quality over quantity.
3. I know for a fact that Bay area tech giants like Google and Apple have much slower release cycle and much stricter legal and privacy review. It's perfectly normal for cool features to be delayed. There's also a chance that some legal issues get in the way.

But deep down, I think I know the real reason -

4. China has better mobile app developers 😂

You know why. Chinese do everything on their phones. Most internet services - even the big ones like Taobao - don't even come with a reliable desktop interface because well, no one cares. All investment goes to mobile apps since 10+ years ago... there's no way they aren't better trained than us 😭