planetary media

Not Film, Not Theater, Not Games, Not AI
Game AI vs. DRL vs. Hybrids vs. FAMs
Artificial Life, Big Worlds, Performance

not film, not theater, not games, not ai

Don't you think we're approaching an entirely new medium that is not film, not theater, not games, not AI, but something that combines all of this together, and it's almost worthy of a new name?

Absolutely, we are indeed approaching a new medium that defies traditional categorizations such as film, theater, games, and AI. This emerging domain combines the interactive and immersive qualities of gaming, the narrative depth of film and theater, and the adaptive, responsive capabilities of AI. It represents more than just a convergence of existing technologies; it's a novel form of expression and experience.

This new field extends beyond a simple combination of existing mediums. It offers a more immersive, interactive, and adaptive form of storytelling and experience, leveraging the strengths of each component while introducing unique elements of its own.

As for naming this new medium, it's common for language to evolve alongside technological and cultural changes. Just as "cinema" was coined for the art of moving pictures and "videogames" for interactive electronic entertainment, this new medium might eventually be defined by a term that encapsulates its distinct blend of technology, art, and interactivity.

This field is still nascent, and its limits and potentials are continually being explored. The term that ultimately becomes associated with it will likely emerge from the community of creators and users who are actively shaping this medium. The name will probably reflect the essence and uniqueness of this convergent form of media.

Can you give the defining features of this medium and what makes it different from all media before it?

Interactivity and Agency: Unlike traditional media like film or theater, where the audience is passive, this new medium allows for active participation. Users can influence and change the narrative or environment, similar to video games but often with more depth and complexity.

Immersive Experience: Leveraging technologies like virtual reality (VR) and augmented reality (AR), this medium offers a level of immersion far beyond traditional screen-based media. Users can feel as if they are inside the story or environment, experiencing it first-hand.

Adaptive Narratives: AI integration allows for narratives that adapt in real-time to user decisions or actions. This is a significant evolution from linear storytelling in films or the fixed paths in most games. The story can change dynamically, offering a personalized experience.

Collaborative and Social Interactions: This new medium often includes social and collaborative elements, allowing users to experience and influence the narrative or environment together, which is a step beyond the isolated experiences of traditional media.

Artificial Intelligence: AI is not just a tool but a fundamental part of this medium. It can create responsive environments, characters with realistic behaviors, or even assist in content creation, like generating visuals or music based on the context of the user's experience.

Real-Time Rendering and Simulation: Technologies like game engines enable real-time rendering and simulation, which allows for dynamic and responsive environments. This is a significant shift from the pre-rendered content of films or the limited dynamics of traditional theater.

Aren't we talking about sleep no more, but in VR with actors and/or agents that allow for the story to change somewhat? Or, minecraft but with intelligent agents? Maybe a broad story people are working through?

Yes, you're right in drawing parallels to experiences like "Sleep No More" and "Minecraft," but with added dimensions facilitated by VR and AI. What we're discussing is an advanced, more dynamic form of interactive and immersive media, which could be thought of as:

"Sleep No More" in VR with AI: "Sleep No More" offers a physical, immersive theater experience where audiences actively explore and interact with a performance environment. Imagine this in a virtual reality setting, where the environment is digitally created and infinitely malleable. AI-driven characters (agents) can interact with participants in a much more complex, responsive manner. The narrative can shift and evolve based on participant actions, making each experience unique.
Enhanced "Minecraft"-like Worlds with Intelligent Agents: "Minecraft" provides a sandbox experience where players create and explore worlds. If we integrate AI into a similar setup, these worlds could become living environments with intelligent agents that can interact, learn, and respond to player actions. This integration would offer a more dynamic and engaging experience, where the environment and its inhabitants evolve organically based on player input and interactions.
Broad, Collaborative Story-Driven Experiences: Imagine a broad narrative that multiple players can participate in, similar to a massively multiplayer online game but with a more profound narrative element. Each player's actions could influence the story's direction, with AI dynamically adapting the plot and world to these choices. This would create a collective but personalized storytelling experience, blending the immersive nature of games with the depth of traditional storytelling.

In essence, we're talking about creating a medium that combines the interactivity of games, the narrative depth of theater and film, and the adaptive intelligence of AI to create a fully immersive, responsive, and dynamic experience. This would allow for storytelling and engagement on a level that's currently unattainable with traditional media.

What are the closest examples?

VRChat: While primarily a social platform, VRChat has seen the creation of user-generated worlds and experiences that blend elements of gaming, theater, and virtual reality. Some of these worlds feature interactive narratives and role-playing elements that hint at the potential of more integrated experiences.
Tender Claws' Projects (including "The Under Presents"): As mentioned earlier, "The Under Presents" is a notable example. It combines VR with live performance, allowing for a unique blend of scripted and improvised interactions. Tender Claws' other projects also often explore the intersection of technology, storytelling, and interactive experiences.
Half-Life: Alyx: This VR game by Valve has been praised for its immersive storytelling and interactive environment. While it doesn’t incorporate AI-driven narrative changes, it represents a significant step in immersive storytelling within a VR environment.
AI Dungeon: AI Dungeon uses AI (specifically GPT-3) to generate dynamic and responsive text-based adventures. It demonstrates the potential of AI in creating adaptive narratives, although it's primarily text-based and doesn’t include VR or live performance elements.
Fortnite’s Live Events: Epic Games' Fortnite has hosted live events within its game world that blend gameplay with storytelling elements. These events are not AI-driven but showcase how a shared, dynamic narrative experience can be presented in a game.
Minecraft with AI Mods: Some Minecraft mods and servers have begun to experiment with AI, whether for enhancing NPC behavior or creating dynamic world events. This illustrates the potential for AI to enhance sandbox gaming environments.

Game AI, DRL, Hybrids, FAMs

What are the types of AI currently used in media like this?

Game AI: This is the traditional form of AI used in games, primarily for controlling non-player characters (NPCs) and game environments. It includes rule-based systems, finite state machines, and behavior trees. Game AI is designed to provide a challenging, enjoyable, and predictable gaming experience.
Deep Reinforcement Learning (DRL) or Imitation Learning: These are more advanced AI techniques. DRL involves training an AI through a system of rewards for achieving desired outcomes in the game environment. Imitation learning involves training AI by having it observe and replicate human actions. Both are powerful but can be challenging to control and direct, as their learning processes can lead to unpredictable outcomes.
Systems Combining Language Models and Game AI: Examples like Voyager and Eureka represent a fusion of Large Language Models (LLMs) with either game AI or DRL. These systems can generate more dynamic and responsive game narratives and interactions, leveraging the conversational and contextual strengths of LLMs.
Foundational Action Models: These are theoretical frameworks designed to apply across a range of games or a specific game. They aim to provide a more generalized approach to game AI, allowing for more versatile and adaptable AI behaviors across different gaming scenarios.
Dialogue LLMs: Tools like Inworld AI focus on creating realistic, engaging dialogue for NPCs. These LLMs can generate contextually relevant and natural-sounding dialogue, enhancing player immersion and interaction quality.

Each of these AI types contributes uniquely to gaming experiences. Traditional game AI ensures predictable and structured gameplay, while DRL and imitation learning bring in a level of unpredictability and adaptiveness. The integration of LLMs with game AI or DRL opens new possibilities for dynamic storytelling and responsive game worlds. Foundational action models aim to standardize and simplify the application of AI in gaming. Dialogue LLMs focus specifically on enhancing communication and interaction within games.

The problem with GameAI is that the behaviors are fixed. Some of the problems with DRL is that they are hard to control. Is a hybrid system like voyager generalizable to other games?

Yes, the basic principles behind a system like Voyager are indeed applicable to other games. The core concept of combining Large Language Models (LLMs) with game AI to create more dynamic, intelligent, and responsive gaming environments is not restricted to any single game like Minecraft. These principles can be adapted and applied across a wide range of gaming genres and platforms. Here's how:

Contextual Interaction: The LLM component of Voyager, which is adept at understanding and generating human-like text, can be utilized in any game that benefits from enhanced dialogue and narrative. This could include RPGs, adventure games, simulations, and more.
Dynamic Content Generation: The ability of Voyager to generate content (like quests, dialogues, or scenarios) based on player interactions is universally applicable. This capability can greatly enhance the replayability and uniqueness of experiences in various games.
Adaptive Gameplay: Integrating LLMs with game AI allows for gameplay that can adapt to player choices and actions in real-time, creating a more personalized and engaging experience. This principle is beneficial in games that emphasize player agency and decision-making.
Enhanced NPC Interactions: Voyager's approach to making NPCs more interactive and responsive can be applied to any game that features NPC interactions, providing a more immersive and realistic experience.
Scalability and Flexibility: The scalable nature of AI and LLMs means that these systems can be adapted to games of different scales, from indie titles to AAA productions.
Learning and Evolution: Over time, systems like Voyager can learn from player interactions, leading to continuous improvement and evolution of the gaming experience. This aspect is particularly valuable in games that evolve or have long-term player engagement.

While the basic principles are broadly applicable, the specific implementation of a system like Voyager would need to be tailored to the mechanics, narratives, and technical requirements of each individual game. This customization is necessary to ensure that the AI integrates seamlessly with the game's existing framework and enhances the player's experience in a way that is coherent and meaningful within the context of that game.

For interesting gaming, don’t the ML approaches to be learning continuously (be online) and isn’t that difficult?

You're correct, many of these advanced AI behaviors in gaming, especially those involving adaptation and learning from player interactions, require "online" learning systems. Online learning in this context refers to the ability of an AI system to learn and adapt its behavior dynamically while the game is being played, as opposed to "offline" learning, where the AI is trained in advance and doesn't change during gameplay.

Implementing online learning in games does present several challenges:

Computational Resources: Online learning can be computationally intensive, requiring significant processing power to continually update the AI's model based on new data. This can be a limiting factor, especially for games intended to run on lower-end hardware or consoles.
Real-Time Performance: The AI must process and learn from new data quickly enough to ensure real-time responsiveness. This is crucial in gaming, where even small delays can disrupt the player experience.
Balancing and Fairness: Continuously learning AI can become unpredictably difficult or easy, potentially leading to unbalanced gameplay. Ensuring fairness and a consistent challenge level is a significant design challenge.
Quality Control and Testing: With AI that evolves during gameplay, quality assurance becomes more complex. The AI's behavior may change in unforeseen ways, making thorough testing challenging.
Safety and Ethical Concerns: Online learning systems can potentially learn undesirable behaviors, especially if they are influenced by player interactions. This raises concerns about the propagation of offensive or harmful content.
Player Experience Consistency: Ensuring a consistent experience for all players can be challenging when the game AI is continually evolving and adapting uniquely for each player.

Despite these challenges, there are ways to mitigate the difficulties:

Hybrid Approaches: Combining pre-trained offline behaviors with limited online learning can provide a balance between dynamic adaptation and predictability.
Simulated Learning Environments: AI can be trained in simulated environments that mimic potential player interactions, reducing the need for real-time online learning.
Restricted Online Learning: Limiting the scope of online learning to specific aspects of AI behavior can reduce computational demands and make the AI's evolution more manageable.
Monitoring and Moderation: Implementing systems to monitor AI behavior and intervene if it learns undesirable patterns can help maintain quality and safety.

In summary, while implementing online learning in game AI is indeed challenging, it's not insurmountable. Advancements in AI technology, along with creative design and engineering solutions, are gradually overcoming these hurdles, leading to more dynamic and responsive gaming experiences.

Will foundational action models (like foundational lanaugage models, but for behavior) ever be a thing?

Foundational Action Models (FAMs) in gaming are an ambitious concept, aiming to create a universal framework or set of principles that can be applied to AI across various games. The development of such models can take two directions:

Game-Specific FAMs: These models are tailored to the unique mechanics, narrative, and environment of a particular game. They focus on optimizing AI behavior within the specific context of that game, ensuring that AI actions are coherent and meaningful within its unique world.
General FAMs for All Games: This approach aims to develop a more universal set of principles or models that can be adapted to any game. These models would identify and abstract the core elements of AI behavior that are common across various genres and styles of games.

The feasibility and development timeline of FAMs, especially the more ambitious universal models, depend on several factors:

Complexity of Game Environments: Games vary greatly in complexity, rules, and player interactions. Developing a one-size-fits-all model is challenging due to these differences.
Advancements in AI and Machine Learning: Progress in AI research, particularly in areas like reinforcement learning, transfer learning, and generalizable AI models, will be crucial in developing FAMs.
Collaboration in the Gaming Industry: Developing universal FAMs would likely require collaboration and standardization efforts across the industry, which can be a complex and time-consuming process.
Ethical and Safety Considerations: As AI becomes more advanced and autonomous, ensuring that FAMs adhere to ethical guidelines and safety standards is essential.

As for a timeline, it's difficult to predict with precision. Game-specific FAMs are already in development in various forms, as many game developers work on advanced AI tailored to their specific titles. However, the development of a universal FAM for all games is a more long-term goal. Given the current pace of AI research and development, we might start seeing the foundational work in this area emerging in the next few years, but a fully realized, broadly applicable universal FAM could be a decade or more away, depending on the aforementioned factors.

artificial life, big worlds, performance

Let's say that these experiences were not taking place in small worlds, but in massive simulated worlds, with towns, creatures, etc. a kind of planetary simulation. People could witness how the world and ecosystem changed over time. A kind of high fidelity version of dwarf fortress. Is that possible?

Creating a high-fidelity, planetary-scale simulation with complex ecosystems, dynamic environments, and detailed interactions, akin to an advanced version of "Dwarf Fortress," is theoretically possible but presents substantial technical and computational challenges. Here’s a breakdown of what such an endeavor would entail and the hurdles that would need to be overcome:

Massive Computational Power: Simulating an entire planet with intricate ecosystems and dynamic interactions in high fidelity would require an immense amount of computational resources. This includes processing power for real-time simulation of environmental changes, weather patterns, ecological dynamics, and individual creature behaviors.
Advanced AI and Machine Learning: AI would need to be sophisticated enough to manage complex ecosystems and individual creature behaviors. This includes AI for procedural content generation, behavior modeling of creatures and ecosystems, and dynamic adaptation to player interactions.
Detailed Procedural Generation: To create vast, diverse, and realistic worlds, advanced procedural generation techniques would be required. This goes beyond terrain generation to include the creation of biomes, weather systems, flora and fauna, and even cultural elements like towns and societies.
Realistic Physics and Environmental Simulation: Accurately simulating physical laws and environmental factors like climate, geology, hydrology, and their interactions would be crucial for a believable world.
Networking and Multiplayer Capabilities: For multiplayer experiences, robust networking infrastructure would be needed to handle the data transfer and synchronization required for many players interacting in a vast, shared world.
Data Storage and Management: Such a simulation would generate a colossal amount of data, requiring efficient data storage, retrieval, and management systems.
User Interface and Interaction Design: Designing an interface that allows players to interact meaningfully with this complex world without being overwhelmed would be a significant challenge.
Sustainability and Scalability: Ensuring that the simulation is sustainable and scalable, both in terms of technology and environmental impact (like energy consumption), is essential.
Ethical Considerations and Content Moderation: With a world so vast and dynamic, content moderation and ethical considerations become complex. The simulation would need to manage these aspects to provide a safe and inclusive environment.
Long-Term Maintenance and Support: A project of this scale would require ongoing maintenance and support to manage the world, update features, and ensure stability.

While current technology is not yet at the point where such a simulation is feasible, advancements in cloud computing, AI, procedural generation, and VR are steadily pushing the boundaries. In the future, with continued technological progress and perhaps new breakthroughs, a high-fidelity planetary simulation of the kind you describe could become a reality. It would represent an unprecedented achievement in gaming, simulation, and interactive media.

I guess what I'm saying is that can you imagine a large world where you can play games in, do performances inside of through motion capture, and it also has its own evolving ecosystem a la artificial life (and/or attempts at open ended evolution).

Yes, envisioning a large, intricately simulated world where players can engage in various activities, including playing games, performing through motion capture, and interacting with an evolving ecosystem akin to artificial life or open-ended evolution, is a fascinating and ambitious concept. Such a world would represent a blend of virtual reality, interactive gaming, digital theater, and complex ecological simulation. Here's how different elements could come together in this envisioned world:

Expansive, Dynamic World: The foundational aspect is a vast, persistent virtual world, rich in detail and variety. This would likely involve advanced procedural generation techniques to create diverse landscapes, ecosystems, and possibly even urban environments.
Interactive Ecosystem and Artificial Life: The world would feature a complex, evolving ecosystem, possibly based on principles of artificial life. This could include simulated flora and fauna that evolve over time, interact with each other and the environment, and respond to player actions. Using concepts of open-ended evolution, these life forms could evolve in unpredictable and novel ways.
Gameplay and Activities: Within this world, players could engage in a variety of games and activities. This could range from structured games with specific objectives to more open-ended exploration, resource gathering, and crafting.
Digital Theater and Performances: Utilizing motion capture and other immersive technologies, players could participate in or create performances. This could be similar to live theater but in a virtual setting, allowing for unique expressions of art and storytelling.
Player Interaction and Collaboration: Players could interact with each other in real-time, collaborate on various projects or performances, and influence the world and its evolving story.
Realistic Physics and Environmental Simulation: The simulation would need to include realistic physics and environmental factors to ensure a believable and immersive experience.
AI-Driven NPCs and Characters: Apart from player characters, the world could be populated with AI-driven NPCs, which could range from human-like characters to fantastical creatures, each with their own behaviors and roles within the ecosystem.
Technological Infrastructure: To support such a complex and dynamic world, a robust technological infrastructure would be necessary. This includes powerful servers for hosting the world, efficient networking for real-time interactions, and advanced graphics rendering technologies.
Accessible Creation Tools: Providing players with tools to create and modify elements within the world, from designing simple objects to scripting complex interactions or ecological dynamics.
Ethical and Safety Frameworks: Ensuring that the world is a safe and ethical space for all players, with systems in place for content moderation, privacy protection, and community management.

Creating such a world is a colossal undertaking and would represent the cutting edge of virtual world design, AI, and interactive technology. While we are seeing the early stages of such possibilities in various separate domains, the integration of all these elements into a cohesive, fully functioning ecosystem is a goal that lies on the horizon of technological advancement. It's a vision that captures the imagination and drives innovation in digital interactive experiences.

I'm imagining that within this big world, you could play/set games, you could make virtual theater, you could make a film, you could do a performance, it would be rich enough that it would be capable of all these things.

Your vision of a vast, multifunctional virtual world where users can engage in a diverse range of activities – from setting and playing games, to creating and participating in virtual theater, making films, and performing – is both innovative and forward-thinking. This concept represents a comprehensive, all-encompassing digital environment that blurs the lines between different forms of media and creative expression. Here's how such a world could function and support these varied activities:

Versatile and Adaptive Game Environment: The world would need to be flexible enough to host a variety of games, ranging from simple, traditional games to complex, immersive experiences. It would require a dynamic system that can adapt to different game rules, objectives, and player interactions.
Tools for Virtual Theater and Performance Art: For theater and performance, the environment would need to support motion capture technology, allowing actors to perform naturally within the virtual space. It would also need tools for scripting, staging, and directing virtual performances, along with the ability to broadcast or share these performances with an audience.
Film Production Capabilities: To facilitate filmmaking, the world would need to offer sophisticated visual and audio recording tools, editing suites, and the ability to control and manipulate the environment and lighting. This would enable directors and filmmakers to use the virtual space as a versatile set for their films.
Rich and Detailed World Design: To support such a wide range of activities, the world itself would need to be rich in detail and expansive, offering a variety of landscapes, settings, and environments. This would provide a suitable backdrop for games, performances, and films.
AI-Driven Characters and Extras: For added realism and interactivity, AI-driven characters could populate the world, serving as NPCs in games, extras in films, or even interactive audience members in performances.
Collaboration and Social Interaction Tools: The platform would need to facilitate collaboration, allowing users to work together in real-time on various projects. This includes communication tools, shared workspaces, and systems for coordinating and managing joint efforts.
Robust Infrastructure for Hosting and Streaming: Given the likely high demand for bandwidth and processing power, a robust technical infrastructure would be crucial. This includes powerful servers for hosting the world and its activities, as well as streaming capabilities for live performances and events.
Content Creation and Modification Tools: Users would need access to a suite of creation tools to design and modify elements within the world. This could range from simple object creation to complex programming for game mechanics or environmental changes.
Intellectual Property Management: With so much creative output, the platform would need a system to manage and protect intellectual property rights, allowing creators to maintain ownership and control over their creations.
Accessibility and Inclusivity: Ensuring that the platform is accessible to a wide range of users, with different abilities and preferences, would be essential for its success.

Creating such a multifaceted and comprehensive virtual world is an ambitious goal that would push the boundaries of current technology. It would require significant advancements in VR, AI, game design, and digital content creation tools. However, as technology continues to evolve rapidly, this vision moves ever closer to becoming a reality. It represents a significant step forward in the convergence of gaming, art, and interactive digital media.

It’s a kind of gaian simulation, but for art.

Your concept of a "Gaian simulation for art" is a compelling and profound idea. Drawing inspiration from the Gaia hypothesis, which views the Earth as a complex, interconnected system, this concept applies a similar philosophy to the realm of art and creativity within a virtual environment. Here's how such a concept could manifest and what it might entail:

Interconnected Creative Ecosystem: Just as the Gaia hypothesis suggests interdependence among organisms and their environment, this virtual world would have interconnected elements of art, performance, and creativity. Actions and creations in one part of the world could influence and inspire developments in another, mimicking an organic, evolving ecosystem.
Dynamic and Evolving Art Forms: The artistic content in this world wouldn't be static but would evolve over time. This could be through direct user interaction or through AI-driven processes that mimic natural evolution, leading to new and unexpected forms of artistic expression.
Collaborative Artistic Processes: Emphasizing the interconnected nature of ecosystems, the platform would encourage and facilitate collaboration among users, leading to collective art projects, performances, and other creative endeavors.
Responsive Virtual Environment: The environment itself would be a canvas for artistic expression, responding to and changing with the artistic activities within it. This could include visual and auditory changes, as well as alterations in the virtual physics or ecology of the world.
Cultural and Biological Diversity: Drawing from the Gaian principle of diversity and symbiosis, the simulation would celebrate diverse artistic expressions and cultural influences, possibly even incorporating AI-generated art that draws from a wide array of human cultures and artistic traditions.
Sustainability and Balance: In line with ecological principles, the platform would aim for a balance between creation and consumption, perhaps using resource management mechanics to encourage sustainable artistic practices.
Educational and Inspirational Potential: This world could serve as a platform for learning about art, creativity, and collaboration, inspiring users to explore new forms of expression and understand the interconnected nature of creativity.
Technological Symbiosis: The seamless integration of various technologies (VR, AI, procedural content generation) would be akin to the symbiotic relationships found in natural ecosystems, each technology supporting and enhancing the others.
Emotionally Engaging Experiences: Just as a healthy ecosystem evokes a sense of wonder and connection, this virtual world would aim to create deeply engaging and emotionally resonant experiences for its users, driven by the power and beauty of collaborative art.
Ethical and Inclusive Framework: Respecting the principles of inclusivity and ethical interactions, the platform would ensure that it is a space safe and accessible for all users, promoting positive and constructive engagements.

Your vision of a Gaian simulation for art represents a groundbreaking approach to virtual worlds, focusing on the organic, dynamic, and interconnected nature of artistic creativity. It's an ambitious and inspiring concept that pushes the boundaries of how we typically conceive of virtual spaces and art. As technology continues to advance, particularly in the fields of AI and VR, such a vision becomes increasingly feasible.