Built in Oculus Quest 2 VR headset, ArtGPT allows you to ask any art history question with your own voice and helps you learn the concept visually.
⌗ Context
Traditional classroom learning experience is not personalized. The teachers need to not only answer all students’ questions but also be familiar with all the details they are teaching. What if each student has a unique question?
▲ The virtual gallery in our app
⌗ Solution
AI tutor is here to help. Whether it’s the Renaissance or the Baroque, ChatGPT has always gotten an answer for you.
With VR, the AI tutor can teach you everything visually no matter where you are.
For example, you can ask “I want to learn about the Impressionism.”
The NPC tutor will tell you what Impressionism is and show you significant Impressionism artworks.
⌗ How we built it
Your question is sent to ChatGPT through the openAI API. We programmed it to generate a list of significant artworks in that period. Then it sources the corresponding images from Wikipedia and presents them in a slideshow in the VR gallery.
At the same time, the prompt is also used to generate a 20 second brief introduction through ChatGPT. The introduction is converted into audio through ElevenLabs. We imported a ReadyPlayerMe character as the NPC tutor, so it looks like a real tutor is talking to you about the topic you just asked.
⌗ Reflection
Before the hackathon began, we settled on developing something related to VR/AR since we all had prior experience with Unity (XR Interaction Toolkit). Coming to a hackathon with a rough idea of something we wanted to build allowed us to jump right in. Thanks to ChatGPT, we were able to figure out the workflow and the steps quickly.
However, getting everything to function correctly in Unity took us quite a bit of time. We ran into issues with controlling the Ready Player Me avatars and generating artworks on separate walls. Due to time constraint, we had to abandon features like first person avatar and synchronizing the tutor’s lecture and body language. Despite the challenges, we were very happy with what we've accomplished in 36 hours overall.
