I want to write my own camera filters for videochat, and ideally apply them in any/all of the popular videochat applications (Zoom, Hangouts, Skype, etc.). The way I imagine this working is to write a macOS application that reads the camera feed, applies my filters, and exposes an additional virtual camera. This virtual camera could then be selected in whichever videochat application.
I've spent many hours researching how to do this and I'm still not clear if it's even possible with modern macOS APIs. There are a few similar questions on StackOverflow (e.g. here, here), but they are either unanswered or very old. I'm hoping this question will collect advice/links/ideas in the right direction for how to do this as of 2020.
Here's what I got so far:
There's a popular tool in the live streaming community called OBS Studio. It captures input from different sources (camera, desktop, etc.), has a plugin system for applying effects, and then streams the output to popular services (e.g. Twitch). However, there is no functionality to expose the stream as a virtual camera on macOS. In discussions about this (thread, thread), folks talk about a tool called Syphon and a tool called CamTwist.
Unfortunately, Syphon doesn't expose a virtual camera anymore: "SyphonInject NO LONGER WORKS IN macOS 10.14 (Mojave). Apple closed up the loophole that allows scripting additions in global directories to load into any process. Trying to inject into any process will silently fail. It will work if SIP is disabled, but that's a terrible idea and I'm not going to suggest or help anyone do that."
Fortunately, CamTwist works. I got it running on my macOS Catalina, applied some of its builtin effects on my camera stream, and saw it show up as a new camera in my Hangouts settings (after restarting Chrome). This was encouraging.
Unfortunately, CamTwist is rather old and not well maintained. It uses Quartz Composer for implementing effects, but Quartz Composer was deprecated by Apple and it's probably living its last days in Catalina.
The macOS SDK used to have an API called CoreMediaIO, which might have been the way to expose a virtual camera, but this API was also deprecated. It's not clear if/what is a modern alternative.
I guess another way of asking this whole question is: how is CamTwist implemented, how come it still works in macOS Catalina, and how would you implement the same thing in 2020?
Anything that sheds some light on all of this would be highly appreciated!
I also want to create own camera filter like Snap Camera.
So I researched around CoreMediaIO and Syphon.
Did you check this Github project?
https://github.com/lvsti/CoreMediaIO-DAL-Example
This repository started off as a fork of the official CoreMediaIO sample code by Apple.
You know, the original code didn't age well since it was last updated in 2012.
So the owner of the repository changed to make it compile on modern systems.
And you can know that the code works in macOS 10.14 (Mojave) to see the following issue.
https://github.com/lvsti/CoreMediaIO-DAL-Example/issues/4
Actually I have not created the camera filter yet because I don't know how to send images to virtual camera that builded by CoreMediaIO.
I would like to know more information. If you know please tell me.
CamTwist uses CoreMedioIO. What makes you think that's deprecated? Looking at the headers in the 10.15 SDK, I see no indication that it's deprecated. There were updates as recently as 10.14.
Is there a good GUI for VR? Scaleform is no longer supported. Coherent seems like an option. I just don't know how to integrate it if it is not natively supported by the engine. IMGUI seems like it is limited in the look that can be created by it. Are there any others?
Have you looked at Qt? It's very flexible and works on a variety of different platforms. QtQuick/QML in particular is pretty nice for VR interfaces.
Here is a video of a talk from Qt World Summit 2017 that does a pretty good job of summarizing the current state of things: https://www.youtube.com/watch?v=RtcmVbicZWw (Basically, the maintainers of Qt are working on a lot of cool VR-related things). I don't work for them and have only starting using Qt recently, but from what I've seen so far it works very well for this purpose.
In particular, take a look at QQuickRenderControl (http://doc.qt.io/qt-5/qquickrendercontrol.html). This class allows full control over rendering of QML content to a framebuffer. Right now I'm using it in the creation of SteamVR overlays.
kudan community. I'm developing an application where I need to use augmented reality, but my application will run on windows computer with two monitors (where one is projecting the augmented reality), connected with a webcamera. Also I need to track an object (not a marker).
I was testing this with the vuforia library and it works on the unity editor, but doesn't work when I try to compile the application on windows. (I know that now vuforia support universal windows apps, but I can't make this works and also from what I see doesn't support multiple monitors).
So I saw that kudan support windows and object tracking but I read the tutorial and wikis and I can't seem make it work.
Any body can guide me on the right track on how achieve this?
PD. I'm using unity engine 5.4 and windows 10 os
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Questions asking us to recommend or find a tool, library or favorite off-site resource are off-topic for Stack Overflow as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it.
Closed 9 years ago.
Improve this question
I've worked for some time with Corona SDK and love how fast and easy I can create powerful apps using Lua. But it can only compile for iOS and Android, which feels like too little now.
My main interest is for it to be able to compile to Desktop AND Mobile. At least for the following:
Windows + Mac for desktop, as standalone applications.
iOS + Android for mobile.
I'd prefer it to lean more towards Lua type scripting instead of ActionScript, but please feel free to post anything that you have worked with and love.
I've found the following engines so far:
Marmalade Quick - After further looking into it, Marmalade Quick can only build for Mobile!
IwGame - Works on top of marmalade and says it can deploy to
desktop and mobile with Lua. Any info is greatly appreciated on this
sio2 - Says "SIO2 is an OpenGLES based cross-platform 2D and 3D
game engine for iOS, Android, MacOS and Windows" and "The engine also
allows you to port your game on the Mac Store and on Windows.", but
their forum and web title is "Game Engine for Mobile Devices". Can't
find any info on if it can deploy to desktop platforms, any info is
greatly appreciated again.
Loom Engine - Loom is similar to Haxe + OpenFL (attempts to attract Flash developers) in that it uses AS3-like of ECMAScript, but it doesn't build native code from it. However it uses Cocos2D for rendering so it should in theory be as fast as Cocos2D. -- Thanks to Bojan.
SDL - I've read in multiple places that SDL can deploy to nearly any platform or device and has a Lua binding. But i can't find how this works as it's not an engine. Any one who can explain how it works and if it's possible is once again, very much appreciated.
SFML - "Windows, Linux, Mac OS X and soon Android & iOS. " doesn't use Lua but can use other languages like Java and Python etc. Anyone have any information on this?
Torgue2D - "Torque 2D was developed with OS X, Windows, and iOS devices in mind and works equally well on all the platforms." uses TorgueScript and no Android =(
Sencha - Seems to compile to all platforms, uses Javascript too which I know. But even with V8 JS would this work well performance wise compared to other options?
GameMaker - own scripting language GML and I actually remember this one as a tool for non-programmers. Has it actually grown into a real engine, I mean for serious development?
Construct2 - Same question as gamemaker
Corona - Lua but mobile only (Android and iOS only as well)
Cocos2D - Seems like it has lots of options but not sure with the same language? Seems like you'd have to re-write your entire code. Any info if cocos2D can deploy to desktop + mobile with almost the same code would be greatly appreciated.
Angel2D - Says it can deploy to everything except Android and uses Lua, anyone ever used this one before?
libgdx --- I've only seen good things about this. Here is a benchmark test for libgdx and is where I saw it reaching 40k sprites at 60fps. http://www.sparkrift.com/2012/1/love2d-vs-allegro-vs-clanlib-vs-libgdx-vs-cocos2d-x-vs-monogame-vs-xna-vs-sfml . It seems libgdx barely goes over 30k actually. But still seems amazing. This is on the same level as Qt for me, almost perfect, except I'm not really worried about performance on it. libgdx can build for everything pretty much.
XNA + MonoGame --- MonoGame's performance seems only slightly lower than libgdx, can build to most platforms. However I don't know much about XNA and I heard it won't be receiving future updates, but is quite stable? More information is welcome.
Citrus --- Don't have much information on Citrus either. AS3 game engine that can build for iOS, Android, Windows, Mac and more.
Haxe + OpenFL --- OpenFL (Haxe) builds to native on many platforms, not just to Flash. Windows, Mac, Linux and Android all get optional native deployment or OpenFL runtime called Neko which is in theory faster than Flash, and SDL 2.0 will enable iOS deployment soon(ish). -- Thanks to Bojan.
Qt-Project --- Just linking Qt project here, can build for everything and has a pretty big community with lots of third party libraries to help you even further.
Moai ---The only Lua engine that I know that can build for Desktop and Mobile. Only downside is the community isn't that big and documentation isn't the best. But if you can get passed those this is a great solution and the one I'm currently using.
Adobe --- Can't forget to add adobe here since it can build to everything that supports flash.
Unity3D --- Recently announced 2D integration looks very promising, should be released Q3-Q4 of 2013.
Cocos2d-x --- An open source engine. Supports JS, Lua, C++ and multiple platforms.
Html5 --- There seem to be a lot of emphasise on html5 mobile apps, here are just a few tools I found that can help port your html5 project to a platform:
Chromium embedded
Sencha
Phonegap
Appcelerator/Titanium
Icenium
So, I'd be happy if you could comment from your experiences with any engines and suggest which one you would recommend.
Thank you for your help!
EDIT: Since this topic is getting popular I'll be adding other options I've found over time. I suggest you choose what is most familiar to you and best for your project needs.
I would recommend V-Play (v-play.net) - it's a cross-platform game engine based on Qt for iOS, Android, Symbian, MeeGo, Blackberry10 and also can export for native desktop applications for Windows, Mac and Linux.
It's based on C++ but has a neat scripting support for QML & JavaScript. QML is a no-brainer to learn and can boost your productivity as less code is needed - just see the comparison with cocos2d-x(60% less Loc) or Corona(15% less LoC) for a comparison of the same games.
(Disclaimer: I'm one of the guys behind V-Play)
If you are into using Python, Kivy is a great solution these days. It compiles to all the platforms you ask for:
Kivy is running on Linux, Windows, MacOSX, Android and IOS. You can
run the same code on all supported platforms. It can use natively most
inputs protocols and devices like WM_Touch, WM_Pen, Mac OS X Trackpad
and Magic Mouse, Mtdev, Linux Kernel HID, TUIO. A multi-touch mouse
simulator is included.
Kivy uses lots of optimized code for graphics rendering (via Cython) so it is fast too.
Here is a speakerdeck that gives you some background and an overview (android specific).
How about HaxeFlixel? We have a great selection of demos, and of course support cross platform development via Haxe + OpenFL. This is an open source project hosted on GitHub. We support all major platforms (including iOS).
Here is my game framework Oxygine.
It is open source modern hardware accelerated 2D C++ framework for mobile and PC platforms.
Features: OpenGL(ES) 2, compressed textures, atlases, complex animations/tweens/sprites, scene graph, fonts, event handling, build tools, and others.
Can be built on top of SDL2 or Marmalade SDK.
In the basis of the engine there is a scene graph, that is similar to Flash one. To be short, You can call this as Flash for C++, but more comfortable and way faster. Initially it was developed for mobile platforms (iOS, Android), but can be also used for PC games.
No mention of App Game Kit (AGK) here so let me fill in the gap. It's a mainly 2D cross platform SDK allowing you to code once in either C++ or it's own "Basic" language. Version 2 just got over 400% funding on Kickstarter and will have full 3D support, Spine support (for 2D animated characters), bullet physics and whole bunch of other new features.
It already has Facebook, Twitter, a bunch of Ultrabook sensor commands, Box2D and more. I've been using it from the start and love it (can you tell?). No, I don't work for The Game Creators (the company that created it) although I admit I did do for a while making some apps.
One of the best features from my point of view is you can develop on Windows and broadcast from the IDE over Wi-Fi to any supported device, so while I'm coding I can (within seconds) test my code on iPad, Android, Windows, Mac or Blackberry Playbook.
If you have C# background. Have a look at Duality.
Duality is a flexible 2D game framework written entirely in C# –
and it’s here to make things a little easier for you. It provides both
an extensible game engine and a visual editor to match. There will be
no need for a level editor, testing environment or content manager
because Duality is all that by itself. And best of all: It’s free.
I'm just answering to give you some insights on how the SDL is used. As you said before it's not a game engine (it's just a library actually). Furthermore, it is not object oriented at all and you don't have some easy animation facilities (you have to code them by yourself).
How it works (I used the C version but I guess the Lua binding should be similar):
Include the headers needed to build the project on the platform you want.
Design your own game loop in which you will set up (at least) a whole event processing system, frame rate manager and a "screen cleaner (or updater)" (I'm insisting on the fact that you have to manually refresh your screen using the SDL_flip_screen routine which is something that is not one of your concerns at all with Corona).
Then, code your game using all the "mechanics" you made before.
The SDL is a low level library (don't expect to have an easy to use GUI framework or the storyboard framework of Corona for instance).
Finally, this library was used to port Civilization III to Linux, so yes it works but it will ask you a lot of energy to have something like you had with Corona ;)
PS: I am not a native English speaker, so please let me know if I wasn't clear :)
Gideros is a great Lua based 2d cross platforms engine, currently supporting both Android and IOS platforms, but more to come.
And it also has some great features as instant on device testing, auto scaling and auto image resolution to easily target various of screen sizes, as well as the option to extend each platform through native plugins.
You also have ShiVa3D, a serious competitor of Unity3D.
It uses Lua and supports many platforms from mobile to game consoles and web browsers.
Very intuitive to use and very nice UI to work with.
There used to be a gui toolkit for experimenting with OpenCV image processing pipelines including code generation. I think it started with an "h". What is it called, and what happened to it?
I think I saw in the past a nice tool to test opencv functions, not so sure about the code generation.
Now i found this one, it looks decent:
http://code.google.com/p/piaf/
This one looks nice but experimental:
http://siggiorn.com/videopipe/
And these, but look poor and ancient:
http://code.google.com/p/cvpreprocessor/
http://code.google.com/p/open-rtvision/
EDIT:
This one is dedicated to feature detecting module:
http://introlab.github.io/find-object/
I think you refer to Harpia. It's available in Ubuntu applications if you enter OpenCV in SW repository (Ubuntu Software Center) search box
This is called HighGUI. You can find many docs in the web. Here's one of them: Load and Display an Image.