My summer research project in the news

From the Penn Engineering website's feature article

Ben Taskar: Machine Learning by Watching and Listening

Ben Taskar is teaching computers how to watch television. Not, as you may think, because they need to relax after reading all that code, but because through this research, Taskar, the Magerman Term Assistant Professor in the Department of Computer and Information Science, is taking machine learning to the next level. Using novel learning algorithms combining video, sound and text streams, his team has shown that computers can be taught to associate what is in a video clip with existing descriptions of characters and actions and then infer information about new material and categorize it according to what it has already learned.

Currently, to categorize videos, photos and other electronic media, computers are “told” through assigned tags the contents of an image. Even new “self-tagging” technologies rely on existing labels to tag new media as it is saved. This is not at all similar to how a human learns and infers. For example, when we watch an episode of our favorite show and we hear one character say to the other, “Joe is coming over soon,” we are able to infer when a new character arrives that his name is Joe. We do not need an explicit label of “Joe” over his face or a subtitle “Joe” at the bottom of the screen.

To expand the boundaries of machine intelligence, Taskar is using television shows with large fan bases like CSI, Alias, and Lost to teach computers how to be smarter about what they see, hear and read. Take, for example, the show Lost. Hundreds of thousands of viewers enjoy spending hours of their time writing and posting scripts of episodes on fan sites, video clips on YouTube, and information in discussion boards. Taskar is taking this collective “wisdom of the crowds” and entering the massive quantities of digitized knowledge and the associated scenes and clips into computers.

From there, computers are given specialized algorithms to be able to combine the information with the video and “learn” which person is which character, what each character is doing, and with whom. At no time does anyone in the research team tag anything. This is known as “unsupervised” or “weakly” supervised learning.

Once this learning has taken place, researchers can ask the computer, “show all scenes where Kate is talking to Jack,” or “produce a montage of all scenes with swimming,” and the computer will generate the sequence. By checking on what is produced, the team then looks for patterns containing errors that suggest the algorithms and models need fine-tuning. Once the algorithm is perfected, the computer can then watch new material and add to the already known information, using its past learning to amass more knowledge.

As you can imagine, using algorithms to teach a computer to learn the nuances of language and parts of speech in written data, along with different camera angles, lighting and other filming conditions, is a daunting task. Taskar compares it to how children learn about their environments. At first, a young child may call all moving vehicles with four wheels “cars,” and later learns to distinguish “trucks” or “vans” from the group. Similarly, computers are given simpler distinctions and tasks at the beginning of learning and more and more complicated ones as patterns to “teach” are better identified.

Future applications of this research go far beyond the “cool” factor of being able to get a computer to show all the scenes in which a favorite character appears. Two areas that will likely benefit are general image and audio search. In order to develop more accurate technologies that can robustly recognize and correctly analyze immense collections of images, videos and spoken language, computers will need to learn to identify hundreds of thousands of different concepts. By tapping into contributions of millions of people on the web and burgeoning data from multiple modalities, the research of Ben and his team will push the field of machine learning towards unsupervised techniques to make computers learn about our complex world.

Computers can only take us so far. We still can’t figure out where those Lost writers are going with that island.

Great is Thy faithfulness

He's always faithful, even when we are faithless.

It's a question of trust, and submission and learning to let go - even when life calls you to take control! To control your own destiny! And to not be a wimp!

But may we obey HIS calling, and not ask Him to make OUR desires and intentions His supposed will.

Been wrestling with this. Uncertainty is always confusing.

Quote of the day, from Pr Isaac, from GK Chesterton: "Jesus promised His disciples 3 things - that they would be completely fearless, absurdly happy, and in constant trouble."

Profound words.

Maybe I've to take back what I said about having shaken off jetlag

Church isn't quite the best place/time to fall asleep (especially when you're sitting in the 3rd row in front of a guest speaker). Or 3-7pm everyday consistently. Or in the car on the way back from church. When you're the driver. Lol! Hahaha I kid - but that's why Chern, Wayne and Daniel are good friends, they remind me that the light is red, or WATCH THERE'S A ROADBUMP.

The changing of the guard in ISCA is evident indeed. Those in F1&2 when I left are now leading, those we used to teach in CIDS are now in ISCA. And thus the challenge is to keep passing down the torch, that we may know God, and make His name known!

And I feel old haha.

Quote of the day: He who abhors durians is a man of God.

Picture of the day: Marcus' mobile phone wallpaper. Ask him to show it to you :p

yayyy

8 months is a lonnggg time. It was a horribly long flight fraught with long delays (and the subsequent frantic running through the airport to catch the next flight), quirky United flight attendants, a very confused internal sleep and eating schedule, but one filled with an eager excitement of finally coming home :) So that made up for everything.

Ring me up - I'm still using the same mobile number! And I'm glad to say I slept 7 good hours last night (even though I crashed all afternoon haha), so I'm hoping the fight against jet lag is over?

Time for a ROTI CANAI breakfast. Mmm.

HOME AWAITS

AM COMING BACK SOON. Yes. I can't wait - today I realized how much I've missed home, and am really excited to be back.

Here are a sprinkling of things that have defined my summer :)

Funny story: We were going on a midnight bubble tea run in Chinatown - we were on the way in a cab, and at a traffic light stop, a couple was making out in a phone booth. Upon my friend's prompt, I opened the window, shouted "I NEED TO MAKE A PHONE CALL", laughed, then shut the window.

Told people stories that made some of my friends jokingly (I hope) tease me that I'm a player back in Malaysia. Their jokes are not grounded on the slightest shred of truth, I assure you. Haha

Played sooo much soccer. I think I've improved immensely in my field sense (haha as opposed to ball control, or stamina etc), probably more than I ever did back home. Which is ironic, because few Americans play soccer - but those who do would have typically undergone daily training in school, hence they are strong at things like through passes/man marking/one touch passing.

Explored my thoughts/emotions and level of trust in God on many different fronts. I think my heart has been tested by God in ways it hasn't been tested before - and I come out of the summer not having any more confidence in my flesh, but knowing that His grace is sufficient, and that He gives wisdom in abundance.

Found out what academic research really is, after alluding to it so much in my college application essays, and discussions about my summer plans. What it really is deserves a whole blog post - but in 3 words, it is: Fulfilling. Challenging. Self-motivated

Rediscovered the wonders of tortilla chips and salsa. And milk and cereal. And wearing flip flops.

Struggled with my reputation and identity - sometimes the desire to be perceived as someone spontaneous and sporting and fun collides with that of being serious and proper and blameless.

Had a 16 year old noob (I WONDER WHO HURHUR) without a girlfriend ask me if I needed help in love. On facebook. For all to see. GG. Upon telling him that a lot of my friends at Penn now wanted to meet him, he asked if they needed help in love too. LOL. Conclusion: Matt needs to step it up hahahahaha

Conclusion #2. Even though Matt loves that 16 year old noob with all his heart, that noob whose name I shall not mention (it's Marcus) better get ready for a beating when Matt gets back. HAHAHA ARE YOU SCARED NOW

See you all in a week :)