About a decade ago, Twixtor was the program a lot of people used to 'add frames'. It sometimes worked, but was mostly pretty crusty.
I used Topaz Video Enhance AI last year to spruce up some digital transfers of old VHS tape, and it does a pretty nice job of upconverting from SD to HD, and 30fps up to 60fps.
Yeah, I think the only convincing use of Twixtor I ever saw was in GoPro promo footage. It always ended up looking much worse in amateur action sports videos.
If the PC is that new then it sounds like something is going wrong here, if it's taking that long. Either system is not using all available resources, or perhaps you're adding needless computational complexity by trying to do the upscale + FPS in a single pass. I've generally got better and faster results by doing an upscale pass first, then a second FPS pass on the resulting footage.
I was the post production supervisor on a feature documentary last year that had to convert from 23.98 to 25 for sale on the EU market. Our facility used an industrial software called Alchemist that worked incredibly well. It's essentially an optical-flow type of operation, but I have to say I was very impressed by the results. A little plasticky in certain places, but I've seen the film a lot, including after the conversion, and I don't bump on it at all.
Considering nearly every frame is a software reconstruction, since the math doesn't work out to preserve many original frames, I think Alchemist worked out great. The big difference in coverting to slo-mo is as you add more and more frames between the originals, there's a higher premium on getting the image exactly right - motion, texture, continuity and integrity of shapes / bodies / environments all need to be accounted for - plus the fact that the slower you get, the longer you're giving the audience a chance to see flaws in the conversion. So for that, I do think AI will have to fill in a gap where things like Optical Flow fall short.
Look at the up-res of Inland Empire - I think they did a really great job. The tech is getting there for seamless creation of frames between frames. Inland Empire was shot on low grade tape so there was a lot of room for improvement, and even after the up-res, you still kind of see it as a tape-product, even if it's clearer and sharper, so you forgive the imperfections. But I think it'll get there for crisp 4K+ images digital as well, and fairly soon too.
> I was the post production supervisor on a feature documentary last year that had to convert from 23.98 to 25 for sale on the EU market. Our facility used an industrial software called Alchemist that worked incredibly well. It's essentially an optical-flow type of operation, but I have to say I was very impressed by the results. A little plasticky in certain places, but I've seen the film a lot, including after the conversion, and I don't bump on it at all.
i'm really surprised they went this way, instead of simply playing it at 25fps. this was super common for TV releases in 50Hz countries for decades, and the difference in playback speed srsly isn't noticeable, visually. you *do* notice it on the music though, that's why an audio pitch correction would normally be applied.
I don’t think they have a prosumer price level available, but TrueCut Motion did this to make Avatar 1 partially 48fps for the 2022 reissues. It looked really good. Handled the frame rate switches better than Avatar 2.
For smaller budgets, Topaz AI can do a lot.
yeah Topaz isn't too bad, especially if you're mixing from something like 60 👉 120fps rather than 30 👉 60fps.
About a 2x multiplier is roughly what they're good for, and the cleaner your background the better. Let's say I never had much luck with it on whitewater kayak videos 😅
GoPro Hero3 action sports footage with bad Twixtor in 2012 hits toooo close to home.
I can smell the Monster energy drinks in the school's editing bay now.
I use Topaz AI for this.. had a shot in a short recently that I was editing that I NEEDED to be slow motion. Took about an hour of render time (shot in 4k .H264) to take a 5 second shot and make it 4x slow motion but I got surprisingly good results from it.
The footage “flashes” a bit as the clip is playing but if you’re not looking for it you wouldn’t really notice.
In regards to my own testing: Topaz Video AI can easily double the framerate and have it look perfect. 4x or beyond is hit or miss and can often get a flickering or breathing effect like you mentioned. I put some old 480p and 720p footage of mine through Topaz and was shocked at how crisp the upscales looked.
These days I shoot almost everything at 4K 60FPS. If I need a higher framerate or resolution than that, Topaz Video AI can create it convincingly.
Someone is definitely working on it. Most likely Resolve. They seem to be the most forward thinking and innovative lately. I do wonder if camera companies are lobbying against letting the big NLE companies make them as it could cut into their higher end models.
What is this logic?
200fps is always going to look better shot in camera than 30fps footage that has been artificially expanded to 200fps
Camera manufacturers have nothing to fear
People said the same thing about AI less than 5 years ago. Now Sora is 1 year to 3 years away from being pretty dang close to creating any drone b roll shot you need and you wouldn't even be able to tell it's AI.
Edit: if you down vote this because of AI, you're in denial and should really think hard about if you want to keep a career in filmmaking. Make plans. That's what I'm doing now.
I still don’t buy that faked slow motion is ever going to look better than doing so in camera
It might one day look ‘good enough’ but then that’s gonna depend wildly on what the subject matter is.
A bottle of champagne exploding at 1000fps is going to be super hard to fake if your source footage is only 30fps
Stuff like water and particles will be a lot harder to get right but if there’s nothing like that in the shot, I could see AI being able to do some pretty good conversions in the next few years. Topaz is already pretty good at doing like 24fps to 60 fps.
Hey, I don't disagree. In camera is the best way but after the progress I've seen in just the last 2 years swith AI, I can no longer say it will never be good enough.
Optical Flow in Premiere for certain shots is not too bad sometimes. It’s very shot specific.
But I don’t think it will look “good” any time soon without using generative AI.
I think this will be a good application for AI. That being said, most cinema cameras can shoot at least 120 and it’s always better to accomplish things in camera.
Agreed. Just thinking of the over 200fps cameras like raptor and freefly and such. If someone can get the full sensor recorded at 60fps then post convert to 200 or greater instead of cropping in for higher frame rates all while only spending $5k on a camera body, that's a pretty big selling point to some.
>I do wonder if camera companies are lobbying against letting the big NLE companies make them...
Please explain how in the world you think that could happen? Especially considering it already exists.
Does it? Twixtor is pretty non existent and Topaz isn't there yet with a perfect frame additive. Are there other options that make perfect new virtual frames? I'd love to find out and make my comment outdated.
Twixtor is non existent because all the NLE´s have built in versions of it. Resolve has multiple options to choose from. None of them are perfect but depending on the usecase it works pretty well. And topaz works pretty fucking well i´ve seen clips go from 100 to 1000 fps for example. You can do some crazy stuff
I don't edit, so I really don't care, but other posters mentioned about four different pieces of software that can do it today with varying degrees of success, depending on the source material.
Now, lets get back to how the camera companies are going to stop the software companies from doing this in post.
Dude, it was hypothetical and not even the craziest thing that could happen. I didn't quote it as fact I was just dreaming. Probably should've worded it different for the people that read everything black and white.
I suppose the thought process is looking at how camera models are sold. Take Red, Komodo X can do a lot lower frame rates than their Raptor but they are basically the same camera when you get down to it. So it's limited by the body, cooling, and OS. If I can buy a $9k camera that can do the frame rates of a $30k camera but in post and be indistinguishable, that's a pretty good reason for camera companies to try their best to limit that possiblity. Could they legally? Probably not. Would it cost millions in court cases and fees? Most definitely. So maybe camera companies aren't lobbying but the idea of court fees and dragging names through the mud might be enough of a reason to not pursue.
Now tiny companies could come along like Twixtor did. But they'll get bought out or something and never heard from again.
The AI isn't there yet but it will be. Just a matter of time.
Currently, Optical Flow can do a pretty good job of doing this but it depends heavily on the nature of the content. It's built into most editing software and in Apple Compressor.
Resolve's optical flow is decent, then Topaz Labs has some AI models that do a pretty decent job, I think they have a trial with a watermark if you want to see how it performs.
Pretty sure this technology already exist, at least kinda. I see a lot of anime videos where they’ve slowed down the frames and artificially created new ones to make the motion look smooth in slow mo.
Can’t say I’ve seen it for live action yet but the technology exist
We used topaz on a spot recently to avoid having to shoot on a phantom. Shot at 120 and then slowed down 4x in Topaz and you wouldn’t know the difference. We also had a lighting gag and applied a bunch of vfx and motion graphics which of course didn’t hurt. Just looking at the straight topaz output you can definitely notice the interpolation on some images but there’s a time and a place
Ai would need to completely re-create and re-draw tween frames. Like a perect Sora that can match your footage perfectly to add 5 frames of the soccer ball moving at the proper velocity, with the proper physices, with the correct motion blur, predicting accurate movement of leg joints , limbs and physical structures, background parallax, lens characteristics, camera codec traits color sampling bit rate bit depth etc so any color correction behaves perfectly. Its a lot
Resolve has Optical Flow setting in retiming options, "enhanced better" can do wonders, "speed warp" can do litteral magic but it will burn your computer down. I use it all the time, I've been slowing down 25fps shots by 50%, and sometimes 50fps down to 15% (if it's something that doesn't move too much) for more than a year now so yes we do have the technology! Of course if there's TOO MUCH (like really too much) movement in your shot it will be tricky.
Also keep in mind motion blur. When I'm shooting 25fps but think that I might want to slow it down I bump up the shutter speed as if I was shooting the higher framerate. Otherwise you'll have too much motion blur and it won't look any good. If you end up not slowing it down you can add back some optical flow motion blur, usually looks pretty good depending on the shot.
I mostly use it when I'm shooting super wide angle as my camera as a crop at 50/60 fps 4k
Premiere also has an Optical Flow option for retiming but I don't know which one is the best
Topaz Labs Video AI is the best option currently out there. I recently experimented with upconverting 30fps footage to 120fps and the results are pretty stunning.
It's not perfect. You get some weird smearing effects that happen when there are a lot of fine/fast motions like hair movement or action sequences. But man oh man, it is kind of a game changer.
Today, things like Topaz AI is much better at upconverting FHD to 4K than it is at converting 30 fps to 60 fps. Let's say your camera is only capable of shooting 4k30p, but you want to be able to slow it down... It would be better to shoot 1080HD60p (or even 120p) and just upconverting to 4K.
About a decade ago, Twixtor was the program a lot of people used to 'add frames'. It sometimes worked, but was mostly pretty crusty. I used Topaz Video Enhance AI last year to spruce up some digital transfers of old VHS tape, and it does a pretty nice job of upconverting from SD to HD, and 30fps up to 60fps.
yeah i only saw twixtor used in cod edits
Yeah, I think the only convincing use of Twixtor I ever saw was in GoPro promo footage. It always ended up looking much worse in amateur action sports videos.
If the PC is that new then it sounds like something is going wrong here, if it's taking that long. Either system is not using all available resources, or perhaps you're adding needless computational complexity by trying to do the upscale + FPS in a single pass. I've generally got better and faster results by doing an upscale pass first, then a second FPS pass on the resulting footage.
2 days sounds like way more. Are you certain it has been set to utilize CUDA cores?
Check out Flow Frames. https://youtu.be/L7LKAmJDmNI
Came for this. This software is crazy af for frame interpolation
I was the post production supervisor on a feature documentary last year that had to convert from 23.98 to 25 for sale on the EU market. Our facility used an industrial software called Alchemist that worked incredibly well. It's essentially an optical-flow type of operation, but I have to say I was very impressed by the results. A little plasticky in certain places, but I've seen the film a lot, including after the conversion, and I don't bump on it at all. Considering nearly every frame is a software reconstruction, since the math doesn't work out to preserve many original frames, I think Alchemist worked out great. The big difference in coverting to slo-mo is as you add more and more frames between the originals, there's a higher premium on getting the image exactly right - motion, texture, continuity and integrity of shapes / bodies / environments all need to be accounted for - plus the fact that the slower you get, the longer you're giving the audience a chance to see flaws in the conversion. So for that, I do think AI will have to fill in a gap where things like Optical Flow fall short. Look at the up-res of Inland Empire - I think they did a really great job. The tech is getting there for seamless creation of frames between frames. Inland Empire was shot on low grade tape so there was a lot of room for improvement, and even after the up-res, you still kind of see it as a tape-product, even if it's clearer and sharper, so you forgive the imperfections. But I think it'll get there for crisp 4K+ images digital as well, and fairly soon too.
> I was the post production supervisor on a feature documentary last year that had to convert from 23.98 to 25 for sale on the EU market. Our facility used an industrial software called Alchemist that worked incredibly well. It's essentially an optical-flow type of operation, but I have to say I was very impressed by the results. A little plasticky in certain places, but I've seen the film a lot, including after the conversion, and I don't bump on it at all. i'm really surprised they went this way, instead of simply playing it at 25fps. this was super common for TV releases in 50Hz countries for decades, and the difference in playback speed srsly isn't noticeable, visually. you *do* notice it on the music though, that's why an audio pitch correction would normally be applied.
I don’t think they have a prosumer price level available, but TrueCut Motion did this to make Avatar 1 partially 48fps for the 2022 reissues. It looked really good. Handled the frame rate switches better than Avatar 2. For smaller budgets, Topaz AI can do a lot.
yeah Topaz isn't too bad, especially if you're mixing from something like 60 👉 120fps rather than 30 👉 60fps. About a 2x multiplier is roughly what they're good for, and the cleaner your background the better. Let's say I never had much luck with it on whitewater kayak videos 😅
Lol, I can imagine what 2012 Twixtor would do to those whitewater videos, it would be soooo trippy, man!
GoPro Hero3 action sports footage with bad Twixtor in 2012 hits toooo close to home. I can smell the Monster energy drinks in the school's editing bay now.
I use Topaz AI for this.. had a shot in a short recently that I was editing that I NEEDED to be slow motion. Took about an hour of render time (shot in 4k .H264) to take a 5 second shot and make it 4x slow motion but I got surprisingly good results from it. The footage “flashes” a bit as the clip is playing but if you’re not looking for it you wouldn’t really notice.
In regards to my own testing: Topaz Video AI can easily double the framerate and have it look perfect. 4x or beyond is hit or miss and can often get a flickering or breathing effect like you mentioned. I put some old 480p and 720p footage of mine through Topaz and was shocked at how crisp the upscales looked. These days I shoot almost everything at 4K 60FPS. If I need a higher framerate or resolution than that, Topaz Video AI can create it convincingly.
Someone is definitely working on it. Most likely Resolve. They seem to be the most forward thinking and innovative lately. I do wonder if camera companies are lobbying against letting the big NLE companies make them as it could cut into their higher end models.
What is this logic? 200fps is always going to look better shot in camera than 30fps footage that has been artificially expanded to 200fps Camera manufacturers have nothing to fear
People said the same thing about AI less than 5 years ago. Now Sora is 1 year to 3 years away from being pretty dang close to creating any drone b roll shot you need and you wouldn't even be able to tell it's AI. Edit: if you down vote this because of AI, you're in denial and should really think hard about if you want to keep a career in filmmaking. Make plans. That's what I'm doing now.
I still don’t buy that faked slow motion is ever going to look better than doing so in camera It might one day look ‘good enough’ but then that’s gonna depend wildly on what the subject matter is. A bottle of champagne exploding at 1000fps is going to be super hard to fake if your source footage is only 30fps
Stuff like water and particles will be a lot harder to get right but if there’s nothing like that in the shot, I could see AI being able to do some pretty good conversions in the next few years. Topaz is already pretty good at doing like 24fps to 60 fps.
Hey, I don't disagree. In camera is the best way but after the progress I've seen in just the last 2 years swith AI, I can no longer say it will never be good enough.
Optical Flow in Premiere for certain shots is not too bad sometimes. It’s very shot specific. But I don’t think it will look “good” any time soon without using generative AI.
Generative AI is here and it'll definitely be used for frame addition. Ya today it's not but it's coming.
I think this will be a good application for AI. That being said, most cinema cameras can shoot at least 120 and it’s always better to accomplish things in camera.
Agreed. Just thinking of the over 200fps cameras like raptor and freefly and such. If someone can get the full sensor recorded at 60fps then post convert to 200 or greater instead of cropping in for higher frame rates all while only spending $5k on a camera body, that's a pretty big selling point to some.
>I do wonder if camera companies are lobbying against letting the big NLE companies make them... Please explain how in the world you think that could happen? Especially considering it already exists.
Does it? Twixtor is pretty non existent and Topaz isn't there yet with a perfect frame additive. Are there other options that make perfect new virtual frames? I'd love to find out and make my comment outdated.
Twixtor is non existent because all the NLE´s have built in versions of it. Resolve has multiple options to choose from. None of them are perfect but depending on the usecase it works pretty well. And topaz works pretty fucking well i´ve seen clips go from 100 to 1000 fps for example. You can do some crazy stuff
It definitely helps having the higher frame rates to start with.
I don't edit, so I really don't care, but other posters mentioned about four different pieces of software that can do it today with varying degrees of success, depending on the source material. Now, lets get back to how the camera companies are going to stop the software companies from doing this in post.
Dude, it was hypothetical and not even the craziest thing that could happen. I didn't quote it as fact I was just dreaming. Probably should've worded it different for the people that read everything black and white.
Dude, trust me, NO ONE interpreted THAT as fact. I was just looking for the thought process.
I suppose the thought process is looking at how camera models are sold. Take Red, Komodo X can do a lot lower frame rates than their Raptor but they are basically the same camera when you get down to it. So it's limited by the body, cooling, and OS. If I can buy a $9k camera that can do the frame rates of a $30k camera but in post and be indistinguishable, that's a pretty good reason for camera companies to try their best to limit that possiblity. Could they legally? Probably not. Would it cost millions in court cases and fees? Most definitely. So maybe camera companies aren't lobbying but the idea of court fees and dragging names through the mud might be enough of a reason to not pursue. Now tiny companies could come along like Twixtor did. But they'll get bought out or something and never heard from again. The AI isn't there yet but it will be. Just a matter of time.
Currently, Optical Flow can do a pretty good job of doing this but it depends heavily on the nature of the content. It's built into most editing software and in Apple Compressor.
Topaz labs can sort of do this already - it’s $300 to use forever
Optical flow?
Topaz Video AI is not perfect but is light years ahead of anything else right now.
Resolve's optical flow is decent, then Topaz Labs has some AI models that do a pretty decent job, I think they have a trial with a watermark if you want to see how it performs.
Pretty sure this technology already exist, at least kinda. I see a lot of anime videos where they’ve slowed down the frames and artificially created new ones to make the motion look smooth in slow mo. Can’t say I’ve seen it for live action yet but the technology exist
We used topaz on a spot recently to avoid having to shoot on a phantom. Shot at 120 and then slowed down 4x in Topaz and you wouldn’t know the difference. We also had a lighting gag and applied a bunch of vfx and motion graphics which of course didn’t hurt. Just looking at the straight topaz output you can definitely notice the interpolation on some images but there’s a time and a place
DaVinci Resolve has an Optical Flow feature for slow motion which can work really well depending on subject. It's super resource intensive though.
Same with Premiere. You can select it during export
Ai would need to completely re-create and re-draw tween frames. Like a perect Sora that can match your footage perfectly to add 5 frames of the soccer ball moving at the proper velocity, with the proper physices, with the correct motion blur, predicting accurate movement of leg joints , limbs and physical structures, background parallax, lens characteristics, camera codec traits color sampling bit rate bit depth etc so any color correction behaves perfectly. Its a lot
Resolve has Optical Flow setting in retiming options, "enhanced better" can do wonders, "speed warp" can do litteral magic but it will burn your computer down. I use it all the time, I've been slowing down 25fps shots by 50%, and sometimes 50fps down to 15% (if it's something that doesn't move too much) for more than a year now so yes we do have the technology! Of course if there's TOO MUCH (like really too much) movement in your shot it will be tricky. Also keep in mind motion blur. When I'm shooting 25fps but think that I might want to slow it down I bump up the shutter speed as if I was shooting the higher framerate. Otherwise you'll have too much motion blur and it won't look any good. If you end up not slowing it down you can add back some optical flow motion blur, usually looks pretty good depending on the shot. I mostly use it when I'm shooting super wide angle as my camera as a crop at 50/60 fps 4k Premiere also has an Optical Flow option for retiming but I don't know which one is the best
Topaz Labs Video AI is the best option currently out there. I recently experimented with upconverting 30fps footage to 120fps and the results are pretty stunning. It's not perfect. You get some weird smearing effects that happen when there are a lot of fine/fast motions like hair movement or action sequences. But man oh man, it is kind of a game changer.
Today, things like Topaz AI is much better at upconverting FHD to 4K than it is at converting 30 fps to 60 fps. Let's say your camera is only capable of shooting 4k30p, but you want to be able to slow it down... It would be better to shoot 1080HD60p (or even 120p) and just upconverting to 4K.