As a little side-project, I have been working on putting the artificial neural networks of
AI Gigapixel to the test and having them upscale another favorite thing of mine... Star Trek: Deep Space Nine (DS9).
The State of Deep Space Nine
Just like Final Fantasy 7, of which I am upscaling the backgrounds, textures, and videos in
Remako mod, DS9 was also relegated to a non-HD future. While the popular Original Series and The Next Generation were mostly shot on film, the mid 90s DS9 had its visual effects shots (space battles and such) shot on video.
While you can rescan analog film at a higher resolution, video is digital and can't be rescanned. This makes it much costlier to remaster this TV show, which is one of the reasons why it hasn't happened.
Remastering Star Trek: Deep Space Nine With Machine Learning
This is where neural networks could come in, I thought. With tools like AI Gigapixel, I knew it might be possible the low definition frames of DS9 can be scaled up to a higher definition such as 1080p or 4K. It would never be the same as proper remastering, but it would a step in the good direction.
So I tried my hand at frame or two, to see what it could do. The results were great. AI Gigapixel uses neural networks trained on real photos. So while it did okay with upscaling the video game renders of Final Fantasy, it did amazing upscaling real-life footage and the bigger budget CGI effects of DS9.
Here are some examples below:
Original 480p definition (click to enlarge)
DS9 Enhanced 1080p definition (click to enlarge)
Original 480p definition (click to enlarge)
DS9 Enhanced 1080p definition (click to enlarge)
These still frames showed promise. In the first set of images, the maintenance crewmen in their spacesuits were nothing more than a few pixely blobs. The upscaling process turned the blobs into much more defined figures
The close-up of the hand also improved. The creases and folds of the fingers and hand look much more detailed, and the baseball really shows off its sheen and the intricate stitching.
Moving Images
The real test, however, was going to be if the upscaling process held up with a sequence of frames aka as a video. Would there be artifacts or other unsightly issues? AI Gigapixel was after all made for upscaling single images so it wouldn't take into account the relation between the individual frames of a moving image.
So I set about upscaling a portion of an episode. I settled on the season six episode: "Sacrifice of Angels". A great Dominion War episode that had both epic space battles and more personal face-to-face moments. And so I set out to work.
I will go into greater detail about my process in a future blog post, but it took me about two days to get everything extracted, upscaled and put it back together in a way that was pleasing. This resulted only in the first five minutes of the episode being done (the episode recap, the opening scene, and the intro). Still pretty good time for a mid-to-high end PC with software that isn't just available to professionals.
The result left me pretty awestruck. It looked better than I had hoped. No weird issues or anything. It looked pretty much like an HD version of DS9. Since (moving) pictures are worth more than a thousand words, here are two comparison videos that show off the improvement I was able to get with this machine learning based upscaling technique.
The first shows off the before-and-after situation with still frames.
The second puts the two videos side by side. Take note how much clearer and sharper the Enhanced version looks.
I highly recommend watching all these videos through your YouTube app on your TV if possible. It gives you more of a sense of how it would feel watching an enhanced DS9 on your TV.
Comparisons are all well and good, but what does it look like if you were to watch it normally? Below is a video of the first five minutes of the episode in full 1080p:
What About 4K?
Honestly, I don't know. While I can upscale the image to a 4K resolution, I don't have a TV or monitor with a 4K native resolution to see if it looks better. I have nonetheless made this video and I am interested to hear from people with people with 4K equipment if it looks better over the 1080p version of the intro.
This nearly melted my computer, as it is a lot more intense to upscale than 1080p so I'll stick to this single video for 4K examples of DS9 Enhanced.
What's Next?
Since I do not own DS9, I can not just do what I want with it. While I would love to release full episodes, this is just not legally possible. These videos serve more as a proof of concept for CBS to look into machine learning and neural networks to help remaster DS9 and move it a bit closer to the HD era.
Imagine what a real team could do, with more powerful equipment, custom trained neural networks (perhaps training the network on TNG vs. TNG Remastered images) and access to the original SD files instead of a DVDRip like me.
What I will do is go into further detail about my process, which will be the subject of a future blog.
Let me know what you think.
