Overview
The user is prompted to strike a specific body pose to access the corresponding memory storage on the computer. Then the system recognizes the pose and retrieves the related stories.

Tools
Adobe After Effect,
Adobe Audition
Machine Learning
p5.js
Procreate

Timeline
May 2023

Concept

The user is asked to make a body pose to access the associated momry storage in the computer. Tools used include p5.js (JavaScript), machine learning models, Adobe After Effect, Adobe Audition, and Procreate.

Motive

Rather than relying on traditional mouse and keyboard navigation to search through folders, I sought to create a more meaningful, interactive, and personal experience for accessing old memories, emphasizing their deep connection to personal history.

Next, I used these drawings to create short animations using Adobe After Effect.

With five stories I wrote, I created narrative stories using Adobe Audition by blending my recorded voice with various sound effects.

Lastly, I built a code on p5.js using a PoseNet machine learning model that allows real-time human pose estimation. I trained the program to recognize five different poses and linked each pose to narration and an illustration I created. The program can classify the pose using a webcam and link directly to related visuals. I also added short animations at the start and end of the program.

Challenges & Accomplishments

The most technically challenging accomplishment was implementing a feature to freeze the canvas when audio is playing. Since PoseNet, the machine learning model, runs continuously and mirrors the user in real-time, it doesn't stop unless the program is terminated.

To address this, I created a new variable, freeze, which ensures the draw function only executes when audio is not playing. Additionally, I learned about the onended() event, which triggers a specified function when the audio or video finishes playing. I used this event to unfreeze the freeze variable, allowing the program to seamlessly repeat the entire process with different audio tracks.

User Guide

First, turn on Mac by pressing the button ‘Turn on your Mac’.
A short animation will appear showing the opening of the folder.
Press the button ‘Search the folder’, and you will see yourself on the screen with five poses on the left side. Choose and follow one of those poses, and you will be able to open up the related illustration and narration of the story.

After you try all five poses, press the third button ‘Close the folder’ to
end the sarch. You will see another animation that closes
the folder and returns to the Mac desktop.