From Trends to Experiences: Co-Creation with Generative Artificial Intelligence in Developing Interactive Multimedia Applications
DOI:
https://doi.org/10.19153/cleiej.27.1.6Keywords:
Generative Artificial Intelligence, Prompt Engineering, Multimedia Development and Production, Interactive Multimedia Applications, Co-CreationAbstract
The emergence of generative artificial intelligence (GenAI) tools has ushered in new possibilities for creating diverse content, encompassing text, images, sound, videos, 3D objects, and even code for programming languages. Within this dynamic landscape, both significant challenges and opportunities arise in the field of interactive multimedia application development. It is imperative that the evolving trends in the utilization of these tools metamorphose into experiences that not only reflect but also facilitate the replication of models, methods, and development approaches. This paper contributes to the academic discourse by presenting three distinct development experiences. Each experience represents the initial steps of an incremental exploration process, thus contributing to exploratory research in this domain. These experiences are presented as case studies, with the first one analyzed retrospectively, and the latter two deliberately designed to scrutinize the challenges and opportunities associated with integrating generative artificial intelligence into the software development process, playing a significant role in a co-creation process. The results presented are qualitative, with detailed independent analyses for each case study, offering a comprehensive description of the findings. Thus, this paper presents a set of recommendations facilitating the transfer, replication, and improvement of these experiences by other multimedia development labs or teams.
References
F. García-Peñalvo and A. Vázquez-Ingelmo, “What do we mean by genai? a systematic mapping of the evolution, trends, and techniques involved in generative ai,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 8, no. 9, 2023.
J. Stonier, L. Woodman, M. Alshammari, R. Cummings, N. Dad, A. Garg, A. G. Busetto, K. Hsiao,
M. Hudson, P. J. Singh, D. Kanamugire, A. Kapoor, Z. Lei, J. Lu, E. Mizouni, A. O. Lungati, M. P. C.
Loebel, A. Sethumadhavan, S. Telford, S. Sarin, K. Bettinger, and S. Teeuwen, “Data equity: Foundational concepts for generative ai,” in arxiv/Computer Science/ Computers and Society. arxiv, 2023.
L. Wolf, T. Farrelly, O. Farrell, and F. Concannon, “Reflections on a collective creative experiment with genai: Exploring the boundaries of what is possible,” Irish Journal of Technology Enhanced Learning, vol. 7, no. 2, pp. 1–7, 2023.
M. A. E. A. Omer, Artificial Intelligence (AI) and Cheating: The Concept of Generative Artificial Intelligence (GenAI). IGI Global, 2024.
A. Nguyen-Duc, B. Cabrero-Daniel, A. Przybylek, C. Arora, D. Khanna, T. Herda, U. Rafiq, J. Melegati, E. Guerra, K.-K. Kemell, M. Saari, Z. Zhang, H. Le, T. Quan, and P. Abrahamsson, “Generative artificial intelligence for software engineering – a research agenda,” in arxiv/Computer Science/Software Engineering. arxiv, 2023.
T. K. F. Chiu, “The impact of generative ai (genai) on practices, policies and research direction in education: a case of chatgpt and midjourney,” Interactive Learning Environments, vol. 0, no. 0, pp. 1–17, 2023.
A. Bozkurt, “Unleashing the potential of generative ai, conversational agents and chatbots in educational praxis: A systematic review and bibliometric analysis of genai in education,” Open Praxis, vol. 15, no. 4, pp. 261–270, 2023.
——, “Genai et al.: Cocreation, authorship, ownership, academic ethics and integrity in a time of generative ai,” Open Praxis, vol. 16, no. 1, pp. 1–10, 2024.
P. Ghimire, K. Kim, and M. Acharya, “Opportunities and challenges of generative ai in construction industry: Focusing on adoption of text-based models,” Buildings, vol. 14, no. 1, p. 220, Jan. 2024.
M. U. Hadi, R. Qureshi, A. Shah, M. Irfan, A. Zafar, M. B. Shaikh, N. Akhtar, J. Wu, S. Mirjalili et al., “A survey on large language models: Applications, challenges, limitations, and practical usage,” TechRxiv, vol. 1, 2023.
B. Min, H. Ross, E. Sulem, A. P. B. Veyseh, T. H. Nguyen, O. Sainz, E. Agirre, I. Heintz, and D. Roth, “Recent advances in natural language processing via large pre-trained language models: A survey,” ACM Comput. Surv., vol. 56, no. 2, sep 2023.
Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, W. Ye, Y. Zhang, Y. Chang, P. S. Yu, Q. Yang, and X. Xie, “A survey on evaluation of large language models,” ACM Trans. Intell. Syst. Technol., vol. Just Accepted, 2024.
H. Dang, L. Mecke, F. Lehmann, S. Goller, and D. Buschek, “How to prompt? opportunities and challenges of zero- and few-shot learning for human-ai interaction in creative applications of generative models,” in arxiv/Computer Science/Human-Computer Interaction. arxiv, 2022.
J. Oppenlaender, “A taxonomy of prompt modifiers for text-to-image generation,” Behaviour & Information Technology, vol. 0, no. 0, pp. 1–14, 2023.
J. Shin, C. Tang, T. Mohati, M. Nayebi, S. Wang, and H. Hemmati, “Prompt engineering or fine tuning: An empirical assessment of large language models in automated software engineering tasks,” in arxiv/Computer Science/Software Engineering. arxiv, 2023.
M. Muller, L. B. Chilton, A. Kantosalo, C. P. Martin, and G. Walsh, “Genaichi: Generative ai and hci,” in CHI EA ’22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, ser. CHI EA ’22. Association for Computing Machinery, 2022.
M. Muller, L. B. Chilton, A. Kantosalo, Q. V. Liao, M. L. Maher, C. P. Martin, and G. Walsh, “Genaichi 2023: Generative ai and hci at chi 2023,” in CHI EA 23: Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, ser. CHI EA 23. New York, NY, USA: Association for Computing Machinery, 2023.
Z. Wu, D. Ji, K. Yu, X. Zeng, D. Wu, and M. Shidujaman, “Ai creativity and the human-ai co-creation model,” in Human-Computer Interaction. Theory, Methods and Tools, M. Kurosu, Ed. Cham: Springer International Publishing, 2021, pp. 171–190.
Y. Lyu, X. Wang, R. Lin, and J. Wu, “Communication in human-ai co-creation: Perceptual analysis of paintings generated by text-to-image system,” Applied Sciences, vol. 12, no. 22, 2022.
C.-Z. A. Huang, H. V. Koops, E. Newton-Rex, M. Dinculescu, and C. J. Cai, “Ai song contest: Human-ai co-creation in songwriting,” 2020.
B. Rabowsky, “Applications of generative ai to media,” SMPTE Motion Imaging Journal, vol. 132, no. 8, pp. 53–57, 2023.
A. Serbanescu and F. Nack, “Human-ai system co-creativity for building narrative worlds,” in IASDR 2023: Life-Changing Design, 2023.
A. Serbanescu, “Human-ai system co-creativity to build interactive digital narratives,” in Multidisciplinary Aspects of Design, F. Zanella, G. Bosoni, E. Di Stefano, G. L. Iannilli, G. Matteucci, R. Messori, and R. Trocchianesi, Eds. Springer Nature Switzerland, 2024, pp. 388–398.
S. Werning, Generative AI and the Technological Imaginary of Game Design. Cham: Springer Nature Switzerland, 2024, pp. 67–90.
J. Urban Davis, F. Anderson, M. Stroetzel, T. Grossman, and G. Fitzmaurice, “Designing co-creative ai for virtual environments,” in Proceedings of the 13th Conference on Creativity and Cognition, ser. C&C ’21. Association for Computing Machinery, 2021.
N. Li, J. Du, V. A. Gonz´alez, and J. Chen, “Methodology for extended reality–enabled experimental research in construction engineering and management,” Journal of Construction Engineering and Management, vol. 148, no. 10, p. 04022106, 2022.
T. Turchi, S. Carta, L. Ambrosini, and A. Malizia, “Human-ai co-creation: Evaluating the impact of large-scale text-to-image generative models on the creative process,” in End-User Development, L. D. Spano, A. Schmidt, C. Santoro, and S. Stumpf, Eds. Cham: Springer Nature Switzerland, 2023, pp. 35–51.
M. Jonsson and J. Tholander, “Cracking the code: Co-coding with ai in creative programming education,” in Proceedings of the 14th Conference on Creativity and Cognition. New York, NY, USA: Association for Computing Machinery, 2022, pp. 5–14.
X. Guo, Y. Xiao, J. Wang, and T. Ji, “Rethinking designer agency: A case study of co-creation between designers and ai,” in IASDR 2023: Life-Changing Design, 2023.
Y. Huang, “The future of generative ai: How genai would change human-computer co-creation in the next 10 to 15 years,” in Companion Proceedings of the Annual Symposium on Computer-Human Interaction in Play, ser. CHI PLAY Companion ’23. New York, NY, USA: Association for Computing Machinery, 2023, pp. 322–325.
J. Zhu, A. Liapis, S. Risi, R. Bidarra, and G. M. Youngblood, “Explainable ai for designers: A humancentered perspective on mixed-initiative co-creation,” in 2018 IEEE Conference on Computational Intelligence and Games (CIG), 2018, pp. 1–8.
G. Allen, G. He, and U. Gadiraju, “Power-up! what can generative models do for human computation workflows?” in arxiv/Computer Science/Human-Computer Interaction. arxiv, 2023.
A. Schmidt, P. Elagroudy, F. Draxler, F. Kreuter, and R. Welsch, “Simulating the human in hcd with chatgpt: Redesigning interaction design with ai,” Interactions, vol. 31, no. 1, pp. 24–31, 2024.
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Alexander Rozo-Torres, Carlos J. Latorre-Rojas, Wilson Javier Sarmiento
This work is licensed under a Creative Commons Attribution 4.0 International License.
CLEIej is supported by its home institution, CLEI, and by the contribution of the Latin American and international researchers community, and it does not apply any author charges whatsoever for submitting and publishing. Since its creation in 1998, all contents are made publicly accesibly. The current license being applied is a (CC)-BY license (effective October 2015; between 2011 and 2015 a (CC)-BY-NC license was used).
