Undergraduate Research at the Vis Center

Prioritizing

Hello again reader,

We’re down to two weeks, and knowing that isn’t much fun.  There’s a lot to do with the project, but little time to do it.  Which means, priorities… it’s time to decide what should come first, so we can finish as many pieces of the project as we can.  With my project, the application using the Kinect, we had considered implementing Image viewers, Video playback, PDF viewers, and 3D viewers.  Ideally, we could have them all by the end of these two weeks.  But ideally is not always reality.

Instantly, I knew the 3D viewer would be the greatest of my worries, since it’s something I’ve never worked with before and I would probably spend the whole two weeks working exclusively on that part.  I had already found something that would help me with the PDF viewer, so that became my first priority.  The nasty side to prioritizing is that when new problems pop up, they can’t be dedicated too much time; interacting with the PDF viewer was troublesome, so I decided to move on to the other two while I figured something out.  I believe I can have all three viewers done soon, especially considering the video and image viewers will be very similar.  Even though it’s a bit of pressure, I’m very excited to finish them and see them all working nicely.

Just a bit of inner discoveries

Hello again, reader

My research project has gone through many phases.  Through those many phases, I’ve learned a lot more about Computer Science.  But more than that, I’ve learned to appreciate it even more.  I used to think you could do anything thanks to computers, but I never thought of myself  as one of those programmers who make those things real.  Now, I think I am more capable.  Either that, or more confident in my abilities and knowledge.

The Kinect was to be used with the 3D projection project, but that idea crumbled when we thought “hey, there’s going to be too much lag, especially since we would have to use two computers”.  Then, I didn’t have much of a goal, but I did have a lot of reading to do to educate myself on Kinect data and its libraries.  Basically, I didn’t have a specific goal in terms of making something, but I did have a goal in understanding the Kinect; learning how to manipulate the information it gave me so I could make something out of it.  That was very exciting, especially since I had to learn not only the Kinect libraries, but how to program in C#.  I learned that I not only like Computer Science, but also greatly enjoy building new things.  This project has helped me realize all this, since I’ve had to learn a whole lot on new topics, have built a library for the Kinect controls, and will be building a simple application.  This project has helped me realize that I can bring my ideas to life, and that I should attempt to bring them to life; it is an immensely satisfying feeling to think up an idea and go through the process of making it real.

Discovery and Challenge

As the fifth week ends, our goals become clearer, and with them, the challenges that must be overcome also become more deeply etched into our thoughts.

Overall, it has become more difficult to advance, as we arrive at deeper topics and obstacles.  Our project has split into two separate projects, since integrating gesture-interaction into the 3D projection system would cause the system to react very slowly to input.  Still, the focus of my research has not changed: How can we create a base system that takes gestures as the main input?  How will displays evolve once these systems become a commonality.  Now, the challenges of delving into this question are clearer.  As it is a new technology, a new form of input for the computer, standard libraries for manipulating the input data are non-existent.

The challenge, then, is that not only must we manipulate and devise intuitive ways in which this new “Skeleton” data interacts with the computer, but we must also write the code which will make the system recognize the data as input of some sort.  Not only must we write the gestures and the system’s reactions to each one, but we must also build, from the ground up, a system that understands that the data received from the Kinect is to be considered input.  After all, skeleton data is just data until we define it otherwise.  These challenges are no small deal, but I will persist and attempt to define a system capable of interacting with the computer easily and intuitively.  I will attempt to build the base on which others may build other structures that will, ultimately, expand to control the entirety of the computer.

These challenges are, at many times, frustrating, but that’s why it’s called research.  Reading and learning and trial-and-error until we get there.

Time flies

Hello once more, reader,

 

They say time flies.  I believe them.  It’s been almost a month and so much has happened, even though it feels like so little time has gone by.  We were given our projects, we adapted to this new environment, we immersed ourselves in the projects, we have learned and read and seen a lot, and we’ve divided the projects further into parts and made them our own, like personal belongings.

When we first arrived, I was clueless, not knowing what I would be doing, or how I would be doing it.  It was a very exciting moment when I was given the 3D projection project for a few reasons.  One of the main reasons was that I didn’t know much about projection, so I knew I was about to learn a whole lot about it; during the first week, it was just enlightening to read the book on projection setups to open my mind to so many bright ideas and to expand my imagination.  In just a week, I was dreaming of projecting stuff everywhere.  The next week, I was even more excited when we were told we’d be given a Kinect to integrate some interaction to our project, and that I’d be working with that part.  It was very of frustrating because I could not get to understand the basics of programming for Kinect with just a few video tutorials on YouTube.  I had ideas, and wanted to bring them to life, but I was getting nowhere… The next week, the pair of books for Kinect programming arrived, and reading the introduction on how the Kinect came to be blew me away.  It was quite interesting how the idea had been developing for around 50 years.  Once I immersed myself in the book, I began to get a better grasp of how to flow with the data and, once more, I had ideas.  This time around, though, I knew I could bring to life some of those ideas, with the help of books.

That’s how we’ve arrived to this fourth week.  Reading, learning, expanding my views, letting my imagination fly.  I feel I’ve grown and defined myself a little more.  I feel as if I have discovered a responsibility that is implicitly part of each of us; I feel it’s almost an obligation not only to imagine, but also to create and build and improve and reinvent.  I feel I’ve found even deeper meaning to one of the quotes I had posted previously on things being based on smaller parts.  I feel we’re going to do great things in what little time we have left, and I feel that we’ve all learned to always strive to research and create.  I don’t know… it’s just a very enlightening experience.

They say time flies, and each of us has so many great ideas…

To create the future…

Hello again, reader,

 

This week, a pair of books on the Kinect SDK arrived, which have been extremely interesting (other than helping with the programming).  I’ve learned a bit of history about how the Kinect slowly came to be: imagined decades ago, different projects slowly building upon the foundation that has made technology like this a reality.  In essence, the Kinect detects motion and sound.  In depth, though, it captures streams of color frames, depth frames and audio strings.  It utilizes that data to produce a 3D virtual environment, and uses other algorithms (which were slowly ‘taught’ to the program) to determine whether a given object at a certain depth is a person, and moves on to determine some of the person’s joints.  It also utilizes not one, but four microphones, which synchronize their information to determine very precisely where the source of that audio is.

 

All of this data combined provides grounds on which programmers can build on; what do they build on it?  Essentially, they build gestures and sounds to which the system reacts; gestures that are intuitive to the users (like sliding a page to a side), words that tell the system what to do.  The real question is how do we expand on it?  How do we make work upon those grounds so that, like the current mouse+keyboard setup, systems like this will be fully functional and ubiquitous?  Imagine a world where, by simply moving around or talking naturally to some system, work gets done.  Imagine entering your office, fully equipped with such a system: the office quickly turns on all the lights, greets you, and begins projecting a calendar near the door, shows you a reminder for an important meeting on the wall you are facing, shows a bunch of contacts scattered on the desks with their respective messages and windows, and just by moving around your arms, you can scan around the contacts to decide which you will look at first, and with a throwing motion, shoot it at the far wall to make it bigger.  Imagine an office that can do that and more.

 

THAT is the future, and we’re closing in on it with this technology.  As we add more sensors to the system, we can make a clearer and more precise virtual environment for the system, making it more effective when reading your actions; as we add break the limits of where we can project images, we get closer to making the entire office a workstation.  As we refine the algorithms used to read the user, the system becomes smarter and reactions are made smoother.  That is a future I want to create.  That is a future I will create.  One single summer is too little time to make this all happen, but it is more than enough time to give life to new ideas, to new goals.

Imagining the Future

During the week, we met up with the guys working on the content for the 3D projection program, and we had a long talk about what could be displayed, how it could and should be displayed, and how to interact with it.  That’s when we decided to introduce the Microsoft Kinect into the project: imagining an entire workspace (or even at home), where interactions with the system are controlled by one’s own movements.  That sent my mind racing… the possibilities, the challenges, the excitement… But, as with everything (and one of my favorite quotes), “Significance is cumulative – but not always obvious.”  Meaning we have to start small, with the bits and pieces that, when merged, will realize that future vision of an interactive motion-sensing space.  Which leads to the focus of the research we will be doing: how do we begin building this smart place that recognizes and reacts to one’s movements?  As is usual when working in a team, each member makes one of those pieces his/her own.

In my case, I’m happy to be working with the interactivity of the Kinect, which produces a research question on a more personal level:

How can we create a basic system that recognizes intuitive gestures and reacts to them accordingly; a system which we can improve later to make it more complete and more interactive?

Week 1 – A new vision

Hello reader,

This first week has been awesome.  I did undergrad research for three semesters, but it’s been a very different experience in basically every aspect of it.  During the first day, Julie took us shopping and to our dorms, so most of the day was getting to know where we’ll be staying at.  Since we had the evening off, we went to see the enormous gym and register ourselves for the summer.

The second day we met a bunch of people (I don’t remember most their names, haha) and were introduced to the projects and which project each of us will be working on.  Both projects were very interesting, and I was sent to the one that I was hoping to get.  It’s based on 3D-projection using non-planar surfaces (we’re starting off with a ball) and multiple projectors to cover a wide area.  We will be in close contact to the fellows on the second floor who are working on a similar project (I’ll learn their names eventually…); we’re going to use the source code of a program they developed to adapt it to what we’ll be doing.  It’s been a very interesting week: as soon as we received the project, we started drawing on the whiteboard trying to figure the least number of projectors we would need to cover the entire sphere.  We were debating on 5 or 6 projectors, but we were given two projectors the next day and through a little trial-and-error figured that the projectors cover a wider area than what we first thought they’d cover.  Now, we think that with as few as 4 projectors should cover the whole thing.  We’ll be working with an engineer next week who will help us set up a workspace to play around with the projectors to figure out the optimal setup.  Meanwhile, we’ve been reading a book titled Practical Multi-Projector Display Design (A. Majumder, M. S. Brown), which has been too interesting.  It’s helped me decide that I will not have TV’s set up in my home (once I actually get my own home, haha), and where there would normally be a TV, there will be, instead, a super cool multi-projector display.  They’re not cheap, so I’m hoping I’ll be successful enough to actually afford that awesomeness.  Go, me!  The book has greatly widened my imagination, in a bunch of fields, not just in terms of displays; there’s a quote from Alan Kay that I liked a whole lot: “The best way to predict the future is to invent it.”

After our shifts in the Marsbury building, we’ve been very busy sightseeing.  We visited the university’s bookstore for souvenirs (and coffee mugs!), we visited the bike shop down Maxwell’s (but they were too expensive, so we’re checking bikes on Craiglist.com), and during the later parts of the evenings, we’ve been spending time in the gym.  Eric’s been busy playing basketball, while Karlo and I have mostly been rock-climbing. Plain. Awesome.

I think that’s about it for the first week.  It’s been very fun and exciting, meeting all the new people, being introduced to this creative environment, exploring new worlds of imagination, staying at a place where so much is so different from what I’m used to… it’s just plain amazing.

eHhP6BEfYAzM