etherealengine / etherealengine Goto Github PK
View Code? Open in Web Editor NEWiR Engine - Bringing us together on the open social spatial web. 🤖 🚀 👓 🕹️ 🧑🏿🚀
Home Page: https://etherealengine.org
License: Other
iR Engine - Bringing us together on the open social spatial web. 🤖 🚀 👓 🕹️ 🧑🏿🚀
Home Page: https://etherealengine.org
License: Other
Skipping auth makes API development a little bit faster since you don't have to juggle tokens or worry about setup, so we should implement this toward the beta milestone.
Identify all points of risk, add auth hooks to routes and define a clear public API surface. Add auth hooks to all services, or set all endpoints to require auth by default and whitelist public routes.
Currently skipping the uploads
service test. Please fix it, since it's currently failing.
We need a UI component for a user to upload an image, video or 3D model to our backend. This will interact directly with our low-level resource API and upload multi-part form data.
We want to be able to switch between the 3D view and a 2D view. This 2D mode allows phones that can't render everything to completely eliminate all the avatars and just focus on the moving experience.
This will necessitate overhead camera and rotate controls.
We use replace avatars with users' badge icons and clip the camera to just above them so our avatars don't get lost under buildings.
What does this feature do?
We want to be able to consume objects and store them, and then make them available when requested.
Actual objects and resources will be uploaded to S3 via our storage API. @barankyle has done this.
First step: Add CRUD for objects. Owner is 'server' (default) unless it is attached to a userID
We already have an Objects API. Update the objects.model.ts to have a list of resources
Feathers automatically makes CRUD -- read documentation to override if necessary
class Object(){
id: number (unique)
owner: relation to user (empty for server)
created: dateTime
modified: dateTime
resource: {type, url}
}
Second step: Integrate with Storage API
AddResourceToObject ({objectId, type, url}) -- Upload new resource and attach it to the object
Third Step: Component on front end that lets user upload a GLTF or GLB (POST)
Fourth step Add image upload, OBJ and FBX upload (POST)
Fifth Step Form shows object, user can edit or replace resource (UPDATE) to delete (DELETE)
Sixth step: Integration with XRPackage. Upload and download XRPackages
Please check out:
https://github.com/webaverse/xrpackage
Continuing on Spoke development, integrating Search API to allow Spoke to search for objects from Sketchfab, Google Poly or internal collections. Media search bar should fully integrate with our backend.
We will eventually want to add a search term blacklist -- this will be handled in a separate ticket, but please design in a way where a search term blacklist is not difficult to build in.
GOAL:
A full stack solution -- API and front end menu -- so users can configure their avatar icon, connect and disconnect accounts and adjust settings.
(Should user settings and user profile be separate / separate related models)
User Badge Menu and Profile Component
In the front end, when you log in, there is a user badge and icon
When you click on the badge, it has a dropdown menu: profile, contacts, logout
If user clicks on profile: open a modal window where user can upload a profile photo (create and embed UserIcon component), edit their settings (create and embed UserSettings component) and connect additional accounts (embed existing "connect accounts" component our other engineer is building)
Details of settings require more review.
Planned so far.
User Icon Component
Upload / replace user icon
This is a "resource" of type "user-icon"
Add resource type to seed-config
Copy /upload-video pattern for user icon upload endpoint
User Profile / Settings Component
Enter room muted: true/false
Microphone volume: 0-100
Audio playback volume: 0-100
Visual Quality: High / Low
Use anti-aliasing: true/false
Our public-video
s Model (and database table) currently looks like:
{
"original_title": "My Video",
"title": "The Video Title to be Displayed",
"link": "https://www.youtube.com/watch?v=abc123",
"description": "Some description",
“thumbnail_url”: “ “something.jpg”,
"production_credit": "...",
"rating": "",
"categories": "...",
"runtime": "..."
"tags": "..."
}
This needs to be moved into the Objects model, since that's exactly what Objects were intended for - external assets with a title, link, description, thumbnail.
(This will be a stepping stone to also storing Scene thumbnails in objects, so we can re-use the video grid component for them.)
(Also, drop the original_title
field - we don't need it.)
Upgrade server to Feathers.js version 3.
Document any upgrade or migration issues.
Let's see if we can fix these warnings on install:
npm WARN deprecated @types/[email protected]: This is a stub types definition. graphql provides its own type definitions, so you do not need this installed.
npm WARN deprecated [email protected]: Feathers v3 is out and this module has moved to @feathersjs/commons. See https://docs.feathersjs.com/migrating.html for more information.
npm WARN deprecated [email protected]: This module has been integrated into @feathersjs/commons.
npm WARN deprecated [email protected]: Feathers v3 is out and has moved to @feathersjs/errors. See https://docs.feathersjs.com/migrating.html for more information.
npm WARN deprecated [email protected]: Feathers v3 is out and this module has moved to @feathersjs/commons. See https://docs.feathersjs.com/migrating.html for more information.
npm WARN deprecated [email protected]: core-js@<3 is no longer maintained and not recommended for usage due to the number of issues. Please, upgrade your dependencies to the actual version of core-js@3.
npm WARN deprecated [email protected]: request has been deprecated, see https://github.com/request/request/issues/3142
npm WARN deprecated [email protected]: Legacy versions of mkdirp are no longer supported. Please update to mkdirp 1.x. (Note that the API surface has changed to use Promises in 1.x.)
Running mediasoup in hosts with private IP (AWS, Google Cloud, Azure)
Those environments run virtual hosts with private IP and provide mechanisms to route an external public IP into that private IP. mediasoup implements ICE-Lite meaning that it won't initiate ICE connections but will always act as ICE “server” role.
In order to run mediasoup in those environments (host with private IP and a mapped public IP):
Let HOST_PUBLIC_IP be the external public IP mapped to your host.
Redirect the port range given by rtcMinPort-rtcMaxPort from HOST_PUBLIC_IP to your host.
Of course, also redirect whichever port your application uses for signaling (HTTP/WebSocket) from HOST_PUBLIC_IP to your host.
Use ip: '0.0.0.0' and announcedIp: HOST_PUBLIC_IP when creating a transport.
https://groups.google.com/forum/#!topic/mediasoup/CwsxtmCcrh4
(This applies to any auto-generated email, but specifically to the Email Login Link emails.)
Email subject line reads "Magic Link to sign in",
Expected: language to mirror the log-in screen and identify the community from which the email is being sent ex: "Login to KaiXR.com"
Right now the email subject line is hard coded, let's instead draw from config.
@mrhegemon has a good idea of what the client wants who's paying for this feature add, so coordinate with him. We are keeping this mainbranch because payments with SSO conneccted to Feathers is already a superhot featureset :)
Rendering just 10 videos at once concurrently on the server is taking a very long time, on the order of an hour. It would likely be better to do this outside of the API server so as not to max it out and cause rendering drops if the API is being redeployed.
Not sure if a standalone server/K8s pod, Lambda, or something else would be the best platform.
Also not sure if doing one at a time from a queue is better than multiple ones at once.
One thing we've considered is giving our server a "mode" type -- so that it can run as a media server and only expose those endpoints, or as an API or realtime gamserver, for example.
In progress?
https://github.com/HydraFire/ecsy-physics-system
We need a physics solution that can be used by all of our networked minigames and locations.
Please use the emscripten port of
NVIDIAGameWorks/PhysX#238
Alternate
ammo.js for physics here -- https://github.com/kripken/ammo.js
Purpose of this feature:
This is like a "party" system in a video game. User A sends user B a request to join User A's party.
This might be expressed as a teacher and a class full of kids. When the teacher presses the button to enter a new world, it will instruct the kids to go with them into this room.
Step 1 -- Create the service (done)
Step 2 -- Associate with users
Step 3 -- Add to party, join party, remove from party and leave party
Step 4 -- Integrate with gameserver and instances for event messages like joining a new server.
Parties are small, ephemeral groups of users. There must be at least two people in a party for it to exist -- otherwise it is destroyed automatically.
Parties have their own chat, and party members can join each other's worlds. Party leaders can invite and kick party members from the party. When a party leader joins a location, every other party member will see an offer to join with them.
Party members can also join the room of any other member at any time, although the intention is that this feature will simplify helping groups all join the same room together.
//@ts-ignore
import blobService from 'feathers-blob';
//@ts-ignore
import fs from 'fs-blob-store';
//@ts-ignore
import S3BlobStore from 's3-blob-store';
Let's look through and find any others and get typedefs made.
Look for any other ts-ignore and let's try to remove them and type anything necessary.
The goal here is a portal that can take a user from one location to another, which implicitly means that it will also be taking them from one instance server to another. This is primarily a front-end component, and the portal-as-a-link concept has been demonstrated, but I think it'd be cool if we didn't something that didn't require reloading Aframe.
My proposed flow:
The void be matrix loading construct, could be a warp gate thing, whatever, have fun with it. Maybe sound?
Purpose of this feature:
Users can take a photo of themselves and add a brief bio.
Synthetic load tests on
static assets
one instance
instance provisioner / agones k8
geo diverse traffic
When our next major deployment is complete and some new server code is up, the following args should be removed, as they will no longer be in use (don't do this until instructed to):
STORAGE_S3_BUCKET_NAME
STORAGE_S3_PUBLIC_VIDEO_PATH
STORAGE_S3_PUBLIC_VIDEO_BUCKET
Goal of this feature:
To route a user from one URL out to one of many different game servers in out Kubernetes cluster.
Two services
location -- creation and deletion of a location, which is a set of instances. Each instance is a game server. Our game servers are manged by Agones -- http://agones.dev
instance -- creation and deletion of instances tied to a location. Each instance is mapped to only one location at a time
Step 1 -- Build the API service and data layer with no API connections
Step 2 -- Add an adapter interface and that be extended by backend adapters for different APIs.
Why? We are currently using a FOSS platform called Mozilla Hubs, and we want to build this feature to work with their stack.
Step 3 -- Build an adapter for our own gameservers. We will need to coordinate with devops and our realtime team on lifecycle hooks.
When deleting a location, we should shut down all active instances for that location as well
Location Model:
id, name, instances (relational to the instance service)
Instance model:
id, location, url, currentUsers, maxUsers
Step 4 -- Add front end components that tap into these APIs.
More stuff, mostly from before the ticket was rewritten:
Location Manager can create a location
Has two adapters: Hubs and XRChat
Optionally can supply a number of instances to allocate to location
For Hubs, this will tap their API to create rooms and return room addresses
For XR3, will tap the Agones API
Set up hooks to poll # of users (Hubs doesn’t have this)
Logic to route users to non-empty, non-full rooms first
Init instance server with information about scene if it isn’t already part of location
Instances with low users can ask Location Manager to reroute users to more full room
When a user goes to location endpoint, redirect them to address of realtime server
Select objects from our backend or external services (Sketchfab, TurboSquid, Google Poly, etc...)
Currently our SMTP config looks like this:
https://github.com/xrchat/xrchat-server/blob/master/src/services/email/email.service.ts
{
host: process.env.SMTP_HOST,
secure: true,
auth: {
user: process.env.SMTP_USER,
pass: process.env.SMTP_PASS
}
}
Need to:
1) Add a PORT config (default to 2525
).
2) Load the secure
argument from config (right now it's hardcoded to true
. Which is the correct default for deployment, but makes it harder to do Mailtrap.io local testing).
This issue has been changed to cover integration of Agones with the Location / Instance APIs so that we can dynamically get IP addresses to game servers from the k8s cluster
We need a dynamic grid component that can be expressed through our object / entity API
Working with agones, redis, mediasoup, etc.
https://docs.feathersjs.com/cookbook/general/scaling.html#cluster-configuration
When user starts server, "failed to remove" errors appear on start (after ~ 20 seconds). This is because we need "use multi" option but don't have specific sequelize definitions for our junction tables, so they are preventing a multi-delete on the sync-seed in dev mode.
We need to figure out how to enable "multi" for models that aren't explicitly defined so that these errors go away
We want to be able to upload files as resources through our Resource API. Ideally we can handle large files -- so let's use Multer to handle multi-part form uploads and integrate with S3 (or, ideally, a storage service with s3 configuration). Once file is uploaded, store url in resource.
Related to https://github.com/xrchat/xrchat-client/issues/96
Set up 1) error logging, and 2) stream those logs to an aggregator like Splunk or Loggly.
(Related: Exception monitoring, issue xr3ngine/xr3ngine-ops#16.)
Users
Has a Contacts list (their friends)
Has many Avatars
Belongs to many Groups (those are ephemeral though)
Contacts
user_id
contact type (friend, blocked, muted etc)
Groups
group_id (string)
user_id (string)
User_role
current_location
XRAvatar
A 3D Model / skeleton, blob. Multiple standard models/formats. There’ll be a list of public templates, but users can have their own / custom ones.
format_type (VRM model, whatever)
Created_by_attribution
owner
name
thumbnail
XRLocationInstance
A particular location has many Instances, for load balancing users. Users can only talk to other people in the same instance. Instances are dynamically allocated; when you join a location, you’re assigned (round robin at first) to a particular instance. (Think “instance dungeon” in an MMO.)
belongsTo Location
belongsTo Scene (references a scene)
location_id - foreign key
XRLocation (XRSpace from the WebXR spec?)
These are the “rooms” that get listed on the front page (or are unlisted, which you can only access via direct link). UI-wise, users “join” a location, but they’re actually assigned to one of the many instances for that location.
Example: VR Event.
is owned by its creator - a User or a Group. (Unless it’s given to somebody else)
hasMany Instances
name (string, unique) - passphrase-style, like Mozilla Hubs
maxUsersPerInstance (number)
access (string) - public/private access etc
XRObject (Not needed right now)
Is an asset file / image / model.
belongsTo User / Group
many-to-many rel’ with Scene (objects are referenced by many Scenes)
object type (static, to start with - “you put it in the room, it stays there.”)
file format
created by
XRScene
“A user made a thing”. A grouping of objects. A scene is assigned to a Location.
Many-to-many rel’ with Locations.
List of objects (with location, position, location. Object id + transform)
(Ideally conforms with Mozilla Spoke)
ScheduledXRLocation
DateTimeRange (turned on / turned off)
Group
Name
XRScene
Payments
Time
User
Type
Purpose of this feature:
We want users to have a basic friend list and ban list
Step 1 -- Users can add and remove other users and friends
Step 2 -- Users can block and mute other users
Step 3 -- Users can view the profile of other users
Step 4 -- Build front end component to reflect all of the API capabillities.
API controls to perform admin operations on users
Kick / Ban / Mute
Move User to Location
Move User to Instance
WHY?
We want to integrate Avaer's webaverse stack.
Step 1: Research webaverse
Step 2: Look at xrpackage (npm install xrpackage)
Step 3: Look at VRM model format
Step 4: Write document on best way to integrate so we can consume and produce XRPackages with content from our API
Step 5: How should this affect / alter our API?
Step 6: Integrate xrpackage and add endpoints to backend
Storage API allows other services to save and load static files and resources without having to implement a specific storage strategy or API. Which storage type is used should be stored in server config (xrchat-server/config/default.json)
Storage API should have two default storage adapters: local and S3. Local is default, and stores files to a public or static directory.
Related to #64
We want to write an adapter to the location and instance services so that they actually do something with regard to backend logic on our server. This means integrating with the Agones API to gain understanding about users in each instance, number of instances, etc etc. to make decisions on where and how to route users.
Cache video to S3 from YouTube
This will need to be fleshed out -- we need to route content requests through to check if it already exists in our cache.
https://github.com/all-contributors/all-contributors
We want to encourage everyone on the team to be on the git and feel involved, so let's design a contributors list that let's everyone share some credit for building this thing.
This will keep our dependencies up to date (as PRs) and let us stay on the latest stuff in a way we can validate, test and roll back.
Create a react UI component that allows current user to connect and disconnect email, sms and oauth providers from their account. This will go into the user profile component that @AyushiMastro is making.
Right now our services register that they work and exist, but that's it. We should add unit tests to validate the models, especially foreign key dependencies, via CRUD.
Currently, users carry around a long list of connected identities Since users will need to be referenced fairly often, and account connections are going to end up with a ton of blank tables for most users, we need to refactor to have an "identity_provider" table which stores oauth, sms and e-mail.
Identity provider hasOne account type, belongsTo user.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.