In brief
Cyril Diagne's AR app prototype uses BASNet machine learning and OpenCV SIFT to copy a real-world object through a phone camera and paste it into Photoshop, with no manual background removal. It takes about 2.5 seconds to cut and 4 seconds to paste, but the code is currently only a research prototype.
When cropping an image in Photoshop, most people typically rely on the mostly manual "Lasso Tool" or the "Quick Selection Tool", which has some AI assistance, to trace the edges of the desired subject and create a selection. But with this AR mobile app developed in France, users can skip this painstaking background-removal process.
Parisian programmer Cyril Diagne is also an artist and designer. Diagne recently shared on his personal Twitter a demo video of the AR (augmented reality) mobile app he developed, which can actually "cut and paste reality".
You read that right. Cyril Diagne's AR mobile app also incorporates machine learning (Machine learning) technology. It can directly copy a real-world object captured by the phone's camera (a potted plant in the video) as an AR image object and paste it into image-editing software such as Photoshop. As long as the phone's camera is pointed at the computer screen, the AR object appears on the image-editing software's interface after a short while.
No one can be 100% certain that Cyril Diagne's video shows a genuine use of the mobile app; it could merely be a CG demonstration produced with special-effects post-production. However, Diagne is by no means bluffing: the development concept behind the app has been published on GitHub. Diagne says the app is currently at the prototype stage. It combines AR and BASNet machine-learning technology, and the "copy" and "paste" steps are separate, each achieved through a different technical principle.
In simple terms, when a photo is taken with the phone's camera, the BASNet machine-learning-powered AI automatically recognizes the object and cuts it out from the real world as an AR object; this process is called "copy". Next, BASNet is combined with another technology called OpenCV SIFT to synchronize the phone and the computer, so that the user only needs to point the phone at the designated position of the image-editing software displayed on the computer screen to complete the "paste" process.
All of these technologies and operating mechanisms are integrated into the app Diagne designed for his phone, and the result is astonishing. It takes about 2.5 seconds for the phone to "cut out" the AR image object from the real world, and only 4 seconds to paste the object into a Photoshop document.
The code behind the AR cut-and-paste can be downloaded from GitHub, but note that it is currently only a research prototype, not a feature aimed at ordinary consumers. If Adobe were to actually integrate this app into Photoshop in the future, it would be a real boon for users.
Original article URL: https://bit.ly/2YO4cZF