Monday, July 3, 2017

Physically Based Lens Flare

While playing Horizon Zero Dawn I got inspired by the lens flare they supported and decided to look into implementing some basic ones in Stingray. There were four types of flare I was particularly interested in.
  1. Anisomorphic flare
  2. Aperture diffraction (Starbursts)
  3. Camera ghosts due to Sun or Moon (High Quality - What this post will cover)
  4. Camera ghosts due to all other light sources (Low Quality - Screen Space Effect)
Image credits: Die Hard (1), Just Another Dang Blog (2), PEXELS (3), The Matrix Reloaded (4)

Once finished I'll do a follow up blog post on the Camera Lens Flare plugin, but for now I want to share the implementation details of the high-quality ghosts which are an implementation of "Physically-Based Real-Time Lens Flare Rendering".

Code and Results

All the code used to generate the images and videos of this article can can be found on github.com/greje656/PhysicallyBasedLensFlare.



Ghosts

The basic idea of the "Physically-Based Lens Flare" paper is to ray trace "bundles" into a lens system which will end up on a sensor to form a ghost. A ghost here refers to the de-focused light that reaches the sensor of a camera due to the light reflecting off the lenses. Since a camera lens is not typically made of a single optical lens but many lenses there can be many ghosts that form on it's sensor. If we only consider the ghosts that are formed from two bounces, that's a total of nCr(n,2) possible ghosts combinations (where n is the number of lens components in a camera lens)


Lens Interface Description

Ok let's get into it. To trace rays in an optical system we obviously need to build an optical system first. This part can be tedious. Not only have you got to find the "Lens Prescription" you are looking for, you also need to manually parse it. For example parsing the Nikon 28-75mm patent data might look something like this:


There is no standard way of describing such systems. You may find all the information you need from a lens patent, but often (especially for older lenses) you end up staring at an old document that seems to be missing important information required for the algorithm. For example, the Russian lens MIR-1 apparently produces beautiful lens flare, but the only lens description I could find for it was this:

MIP.1B manual

Ray Tracing

Once you have parsed your lens description into something your trace algorithm can consume, you can then start to ray trace. The idea is to initialize a tessellated patch at the camera's light entry point and trace through each of the points in the direction of the incoming light. There are a couple of subtleties to note regarding the tracing algorithm.

First, when a ray misses a lens component the raytracing routine isn't necessarily stopped. Instead if the ray can continue with a path that is meaningful the ray trace continues until it reaches the sensor. Only if the ray misses the sphere formed by the radius of the lens do we break the raytracing routine. The idea behind this is to get as many traced points to reach the sensor so that the interpolated data can remain as continuous as possible. Rays track the maximum relative distance it had with a lens component while tracing through the interface. This relative distance will be used in the pixel shader later to determine if a ray had left the interface.

Relative distance visualized as green/orange gradient (black means ray missed lens component completely)

Secondly, a ray bundle carries a fixed amount of energy so it is important to consider the distortion of the bundle area that occurs while tracing them. In In the paper, the author states:
"At each vertex, we store the average value of its surrounding neighbours. The regular grid of rays, combined with the transform feedback (or the stream-out) mechanism of modern graphics hardware, makes this lookup of neighbouring quad values very easy"
I don't understand how the transform feedback, along with the available adjacency information of the geometry shader could be enough to provide the information of the four surrounding quads of a vertex (if you know please leave a comment). Luckily we now have compute and UAVs which turn this problem into a fairly trivial one. Currently I only calculate an approximation of the surrounding areas by assuming the neighbouring quads are roughly parallelograms. I estimate their bases and heights as the average lengths of their top/bottom, left/right segments. The results are seen as caustics forming on the sensor where some bundles converge into tighter area patches while some other dilates:


This works fairly well but is expensive. Something that I intend to improve in the future.

Now that we have a traced patch we need to make some sense out of it. The patch "as is" can look intimidating at first. Due to early exits of some rays the final vertices can sometimes look like something went terribly wrong. Here is a particularly distorted ghost:


The first thing to do is discard pixels that exited the lens system:
float intensity1 = max_relative_distance < 1.0f;
float intensity = intensity1;
if(intensity == 0.f) discard;

Then we can discard the rays that didn't have any energy as they entered to begin with (say outside the sun disk):
float lens_distance = length(entry_coordinates.xy);
float sun_disk = 1 - saturate((lens_distance - 1.f + fade)/fade);
sun_disk = smoothstep(0, 1, sun_disk);
//...
float intensity2 = sun_disk;
float intensity = intensity1 * intensity2;
if(intensity == 0.f) discard;

Then we can discard the rays that we're blocked by the aperture:
//...
float intensity3 = aperture_sample;
float intensity = intensity1 * intensity2 * intensity3;
if(intensity == 0.f) discard;

Finally we adjust the radiance of the beams based on their final areas:
//...
float intensity4 = (original_area/(new_area + eps)) * energy;
float intensity = intensity1 * intensity2 * intensity3 * intensity4;
if(intensity == 0.f) discard;

The final value is the rgb reflectance value of the ghost modulated by the incoming light color:
float3 color = intensity * reflectance.xyz * TempToColor(INCOMING_LIGHT_TEMP);

Aperture


Image credits: 6iee

The aperture shape is built procedurally. As suggested by Padraic Hennessy's blog I use a signed distance field confined by "n" segments and threshold it against some distance value. I also experimented with approximating the light diffraction that occurs at the edge of the apperture blades using a simple function:


Finally, I offset the signed distance field with a repeating sin function which can give curved aperture blades:


Starburst

The starburst phenomena is due to light diffraction that passes through the small aperture hole. It's a phenomena known as the "single slit diffraction of light". The author got really convincing results to simulate this using the Fraunhofer approximation. The challenge with this approach is that it requires bringing the aperture texture into Fourier space which is not trivial. In previous projects I used Cuda's math library to perform the FFT of a signal but since the goal is to bring this into Stingray I didn't want to have such a dependency. Luckily I found this little gem posted by Joseph S. from intel. He provides a clean and elegant compute implementation of the butterfly passes method which bring a signal to and from Fourier space. Using it I can feed in the aperture shape and extract the Fourier Power Spectrum:


This spectrum needs to be filtered further in order to look like a starburst. This is where the Fraunhofer approximation comes in. The idea is to basically reconstruct the diffraction of white light by summing up the diffraction of multiple wavelengths. The key observation is that the same Fourier signal can be used for all wavelengths. The only thing needed is to scale the sampling coordinates of the Fourier power spectrum:

(x0,y0) = (u,v)·λ·z0 for λ = 350nm/435nm/525nm/700nm

Summing up the wavelengths gives the starburst image. To get more interesting results I apply an extra filtering step. I use a spiral pattern mixed with a small rotation to get rid of any left over radial ringing artifacts (judging by the author's starburst results I suspect this is a step they are also doing):


Anti Reflection Coating

While some appreciate the artistic aspect of lens flare, lens manufacturers work hard to minimize them by coating lenses with anti-reflection coatings. The coating applied to each lenses are usually designed to minimize the reflection of a specific wavelength. They are defined by their thickness and index of refraction. Given the wavelength to minimize reflections for, and the IORs of the two medium involved in the reflection (say n0 and n2), the ideal IOR (n1) and thickness (d) of the coating are defined as n1 = sqrt(n0·n2) and d=λ/4·n1. This is known as a quarter wavelength anti-reflection coating. I've found this site very helpful to understand this phenomenon.

In the current implementation each lens coating specifies a wavelength the coating should be optimized for and the ideal thickness and IOR are used by default. I added a controllable offset to thicken the AR coating layer in order to conveniently reduce it's anti-reflection properties:

No AR Coating:

Ideal AR Coating:

AR Coating with offsetted thickness:

Optimisations

Currently the full cost of the effect for a Nikon 28-75mm lens is 12ms (3ms to ray march 352x32x32 points and 9ms to draw the 352 patches). The performance degrades as the sun disk is made bigger since it results in more and more overshading during the rasterisation of each ghosts. With a simpler lens interface like the 1955 Angenieux the cost decreases significantly. In the current implementation every possible "two bounce ghost" is traced and drawn. For a lens system like the Nikon 28-75mm which has 27 lens components, that's n!/r!(n-r)! = 352 ghosts. It's easy to see that this number can increase dramatically with the number of component.

An obvious optimization would be to skip ghosts that have intensities so low that their contributions are imperceptible. Using Compute/DrawIndirect it would be fairly simple to first run a coarse pass and use it to cull non contributing ghosts. This would reduce the compute and rasterization pressure on the gpu dramatically. Something I intend to do in future.

Conclusion

I'm not sure if this approach was ever used in a game. It would probably be hard to justify it's heavy cost. I feel this would have a better use case in the context of pre-visualization where a director might be interested in having early feedback on how a certain lens might behave in a shot.

Image credits: Wallup

Image credits: Wallpapers Web

Finally, be aware the author has filled a patent for the algorithm described in his paper, which may put limits on how you may use parts of what is described in my post. Please contact the paper's author for more information on what restrictions might be in place.

66 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Great post. :-) Thanks for sharing!


    Regarding this:
    "I don't understand how the transform feedback, along with the available adjacency information of the geometry shader could be enough to provide the information of the four surrounding quads of a vertex (if you know please leave a comment)."

    I may be wrong, but perhaps the author defines the cross section of the beam using a quad patch (bounded by 4 vertices). Therefore, the beam is not centered on a vertex, but rather on the center of the quad, so you don't have to share any information between different quads.

    ReplyDelete
  3. very nice thanks admin good post thanks admin
    baby alive videos

    ReplyDelete
  4. Have you looked into https://graphics.tudelft.nl/Publications-new/2013/LE13/PracticalReal-TimeLens.pdf (and https://placeholderart.wordpress.com/2015/01/19/implementation-notes-physically-based-lens-flares/)?

    ReplyDelete
  5. Microsoft office has a different version, and the entire version has the different setup file. Some of the version is Office 2013, Office 2016 and Office 365 etc. Go to www.office.com/setup and redeem your key. Office Com Setup is one of the most popular productivity programs around. If you have your product key handy, you don't even need a disc to start installing Office just visit office setup help. Norton is one of the most reliable antivirus providers in the market to download visit www.norton.com/setup . There are multiple manufacturers making printers for home and office use with varied range as printer series per the need and affordability of different customers. Printer Tech Support. It has become one of the most useful computer hardware devices that allow people to convert computer generated typescript into a physical printed document.

    ReplyDelete
  6. McAfee provides security for all sorts of users. They supply services and products for home and office at home, enterprise businesses with over 250 workers, and small organizations with under 250 employees, and also venture opportunities.
    mcafee.com activate

    ReplyDelete
  7. We are providing help and support for Microsoft office Setup and activation. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get the MS Office application suite and as per your need and see how it is easy to work with Microsoft Office.

    Office.com setup

    ReplyDelete
  8. This comment has been removed by the author.

    ReplyDelete
  9. With an Office 365 subscription, you get the latest Office apps – both the desktop and online versions – and updates when they’re released.
    office.com.setup
    Norton is one of the most reliable antivirus providers in the market. Norton is delivering top rated protection from mobile devices and computers. Norton has 3 antivirus packages
    norton.com/setup
    Microsoft office setup is the software setup file with this setup file you can install on your computer and some of the supported device to use Microsoft office.
    www.office.com/setup
    McAfee have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.
    mcafee.com/activate
    mcafee card activate

    ReplyDelete
  10. Online Help – Step by Step guide for Norton Setup, Download & complete installation online. We are providing independent support service if in case you face problem to activate or Setup Norton product. Just fill the form and will get in touch with you as quick as possible.

    Norton.com/setup

    ReplyDelete
  11. mcafee.com/activate have the complete set of features which can protect your digital online and offline life of the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer.

    Mcafee.com/activate
    install mcafee

    ReplyDelete
  12. Install Webroot for complete internet browsing & web Security in your computer. It will save you from all the cyber attacks.

    webroot install key code

    webroot install

    ReplyDelete
  13. The webroot antivirus is a very renowned security tool that protect the computer software and malware & firewall.

    webroot download

    secure anywhere

    ReplyDelete
  14. Mcafee.com/activate have the complete set of features which can protect your digital online life the computing devices, and it not only help you to protect it but also it can maintain the stability of your computer, increase the speed with inbuilt PC Optimisation tool.

    mcafee activate product key

    mcafee activate

    ReplyDelete
  15. If you haven't already, you'll need to redeem and install Office on your PC or Mac before you can activate.

    office.com setup

    office com setup

    ReplyDelete
  16. Download the Microsoft office setup 365 & Get full support of any query. Get the complete process of installation & product, the procedures to install MS Office keys etc.

    office.com/setup

    www.office.com/setup

    ReplyDelete
  17. We help in setting up and Installing Microsoft Office on your computer. We provide Technical Support and other services to you. We help you with great work and other details. We are here to help you. We help you with Microsoft Office and Office installation. We help you with Great technical services at an affordable price.
    visit:- www.office.com/setup

    ReplyDelete
  18. great website
    http://www.officesetup.company

    ReplyDelete

  19. Awesome, Thank you so much for sharing such an awesome blog...

    Office 365 support

    Office 365 migration

    ReplyDelete
  20. How you install or reinstall Office 365 or Office 2016 depends on whether your Office product is part of an Office for home or Office for business plan. If you're not sure what you have, see what office.com setup products are included in each plan and then follow the steps for your product.
    office-com setup
    office com/setup
    mcafee.com/activate

    ReplyDelete

  21. It is diffcult to setup canon printer but you no need to worry just visit canon.com/ijsetup and get help by printer experts. This will help you in setup all kind of canon printers by canon.com/ijsetup.

    ReplyDelete
  22. Before you plan to install the Office 2016 or Office 365 on your device be
    it a Computer, Laptop, Mobile Phone or a Tablet, you are required to take
    few important steps on of them is to remove any existing Office
    installations from your PC. Just like the previous Office products, Office
    2016 & 365 will conflict with the previously installed versions. So, it
    becomes necessary to remove the previous office files properly.



    setup.office.com
    office.com
    www.office.com/setup

    ReplyDelete
  23. To Setup retail card please visit official website.
    Office Retail Cards allow you to download your security product from the
    internet instead of installing from a CD, ensuring recent versions.



    www.office.com/myaccount
    office com setup
    www.office.com setup

    ReplyDelete
  24. We are providing help and support for Microsoft office Setup and activation. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get the MS Office application suite and as per your need and see how it is easy to work with Microsoft Office.
    Office.com/setup
    www.office.com setup
    Install Office

    Norton is one of the most reliable antivirus providers in the market. Norton is delivering top rated protection from mobile devices and computers. Norton has 3 antivirus packages
    norton.com/setup

    ReplyDelete

  25. with your entire process to setup office product online. Have you Just bought Microsoft Office product ? If yes then you can complete your Office Setup online with your product key code. You just need open office.com/setup , Install Office , Install Microsoft Office or setup.office.com into your web browser.


    office setup
    www.office.com/setup
    office.com/setup

    ReplyDelete

  26. In case any message saying ‘Stop’ pops up while installing Microsoft Office 365 due to a compatibility issue, contact our live chat support.Our online Experts through live chat will guide you through the entire process of Office setup, covering all steps and issues.Keep your 25 characters long product key with you.




    Office Com Setup
    office com/setup
    www.office.com/setup

    ReplyDelete

  27. install office setup. it's very useful for business,school,home.Install Office Setup
    Office com Setup To get started with your Microsoft Office Installation you must need valid product key code and we can also help you with your entire process to setup office product online.

    ReplyDelete

  28. Install Mcafee antivirus to protect your computer or laptop from virus attacks. Mcafee Activate
    Visit there to activate mcafee antivirus with activation code.

    ReplyDelete

  29. Microsoft Office is an office suite that includes a variety of applications, servers and services.ms office setup
    These includes well-known programs such as Word, Excel, PowerPoint, Access, Outlook, OneNote, as well as applications such as Publisher, Project, Skype for Business, Visio, and SharePoint Designe

    ReplyDelete

  30. Office Setup is available for all platforms with different versions like Office setup 2016, Office setup 365, Office setup 2013, Office setup Home & student, Outlook, Office com setup, Microsoft Office Setup, office.com/setup
    http://officesetup.tech/

    ReplyDelete

  31. Install mcafee product key primarily develops digital-security tools for personal computers and server devices.Install mcafee product key this is the good way to protect your pc.

    ReplyDelete

  32. Norton setup product key installation is a computer program that provides malware prevention and removal during a subscription period and uses signatures and heuristics to identify viruses. Norton setup product key installation Other features included in the product are a personal firewall, email spam filtering, and phishing protection.

    ReplyDelete


  33. Office Setup is available for all platforms with different versions like Office setup 2016, Office setup 365, Office setup 2013, Office setup Home & student, Outlook, Office com setup, Microsoft Office Setup, office.com/setup


    www.office.com/setup

    ReplyDelete

  34. mcafee product activation is a antivirus programme. this programme secure your computer with virus so you can install it.mcafee.com/activate Visit mcafee.com/activate to installmcafee with activation code.

    ReplyDelete
  35. Bitdefender offers a suite of Antivirus products that includes Antivirus Plus, Internet Security, and Total Security.central.bitdefender.com Using the Bitdefender tools, users are protected from most online threats including spyware, malware, viruses and other malicious software.

    ReplyDelete

  36. office.com/setup is the brand name Microsoft uses for a group of subscriptions that provide productivity software and related services.office.com/setup if you are getting any error in intallemnt of office.com/setup please visit our website.

    ReplyDelete

  37. Collaborate for free with online versions of Microsoft Word, PowerPoint, Excel, and OneNote. Save documents, spreadsheets, and presentations online, in OneDrive.

    http://office-my-account.net/


    ReplyDelete

  38. Collaborate for free with online versions of Microsoft Word, PowerPoint, Excel, and OneNote. Save documents, spreadsheets, and presentations online, in OneDrive.


    office my account

    ReplyDelete

  39. Collaborate for free with online versions of Microsoft Word, PowerPoint, Excel, and OneNote. Save documents, spreadsheets, and presentations online, in OneDrive.

    office my account


    ReplyDelete

  40. install setup.office.com its useful your busniess,home,school.its popular suite consisting applications, servers and services.setup.office.com The basic version of Office included only Word, Excel and PowerPoint.please visit our website.

    ReplyDelete

  41. Microsoft Product Activation is a DRM technology used by Microsoft Corporation in several of its computer software programs, most notably its Windows operating system and its Office productivity suite.mcafee activation code it's large use in business purpose.

    ReplyDelete

  42. Antivirus software such as mcafee activation code SecureAnywhere Antivirus frustrates malware in a couple of ways. It scans data and blocks viruses that it detects. And it removes malware that is already lodged in a computer.mcafee activate You can tell it to scan your PC according to a schedule that you select.

    ReplyDelete
  43. Office includes word processing, spreadsheet, presentation graphics and email communication programs that provide functionality.
    www.office.com/setup

    ReplyDelete
  44. AOL Email Support is one of the best email service providers that helps in AOL email related issues.There are several email providers like gmail, Hotmail, rediff and so on but AOL is the only company that provides the best service all over the world. aol related issues.
    aol email support

    ReplyDelete
  45. webroot is the best spyware blocking and removel product called webroot spy sweeper. webroot also provides mobile device security.
    www.webroot.com/safe

    ReplyDelete
  46. yahoo is search engine and web service provider.its also known for its web portal.yahoo provides business services also like yahoo web hosting.
    yahoo support

    ReplyDelete
  47. quick book is an accounting software for small and medium sized businesses where we can manage business payments, pay bills and pay rolls
    quickbook support

    ReplyDelete
  48. Microsoft office provides quality of products like word, excel, power point, access, outlook, one drive etc
    www.office.com/setup

    ReplyDelete
  49. Norton is the One of the best antivirus company all over world. It is an anti malware software developed and distributed by Symantec corporation.Norton setup uses heuristics and signatures to detect viruses in your computer. It features also include phishing protection and email spam filtering .Norton setup product is distributed as box copy, a download and as an OEM software.
    norton.com/setup

    ReplyDelete
  50. McAfee have the complete set of features which can protect your digital online offline computing devices that increase the speed with inbuilt PC Optimization tool.
    mcafee.com/activate

    ReplyDelete


  51. Microsoft office setup has various product for various purpose.

    Office Com Setup

    ReplyDelete


  52. Microsoft Office Installation you must need valid product key code & visit office com/setup and we can also help you with your entire process to setup office product online.



    install-office

    ReplyDelete
  53. Hi there! I simply want to offer you a huge thumbs
    up for your excellent info you have here on this post.
    I am coming back to your website for more soon
    Warm Regards www.office.com/setup

    ReplyDelete
  54. Microsoft Office is very necessary productivity software developed by the multinational technological company Microsoft. Microsoft office/ MS Office is developed by the Microsoft. The software was first introduced on 1st August 1988 by Bill Gates in Las Vegas. When Microsoft office was first introduced, the first version of it contained Microsoft Excel, Microsoft Word, and Microsoft Power point. After few years of the release of Microsoft Office become a business software and billion of people are using this by downloading the Microsoft Office setup. Call us : US : +1-888-254-4408 UK : +44-0808-234-2376 AUS : +61-1-800-985-062
    http://officecomsetup.org

    ReplyDelete
  55. We are providing help and support for online activation,you can get required help from office.com/setup. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get complete suite ready and tested as per your need and see how it is easy to work with Microsoft Office.Call us : US : +1-888-254-4408 UK : +44-0808-234-2376 AUS : +61-1-800-985-062
    http://officesetup.ca

    ReplyDelete
  56. We are providing help and support for online activation,you can get required help from office.com/setup. Call us or email us the error or problem, our one of the expert contact you with the suitable perfect solution. Get complete suite ready and tested as per your need and see how it is easy to work with Microsoft Office.Call us : US : +1-888-254-4408 UK : +44-0808-234-2376 AUS : +61-1-800-985-062
    http://officesetup.ca

    ReplyDelete
  57. The Norton Products for which we give underpins are Norton Security, Norton 360, Norton Internet security, Norton Software for Mobile, Norton Deluxe, Norton standard and other Norton Products. You can introduce them utilizing a CD Drive or Online from you dashboard of Norton account. In any case, before that, you need to download Norton Setup from norton-installsetup.com. Pre-necessities of framework and establishment esteem differentiate from Norton thing to thing so fastidiously read the assistance and faq accessible on the official site page of Norton, norton.com/setup. Call us : US : +1-888-254-4408 UK : +44-0808-234-2376 AUS : +61-1-800-985-062
    http://nortoncomsetupnorton.net

    ReplyDelete
  58. The Terms and Conditions thusly is also basic open all the extra conditions germane for the individual set up picked by you. These conditions ar fundamental general acumen. For misuse our affiliations it's relied on to see these 'Terms and Conditions' work as a fiddle. If, you don't fit in with any of those conditions, by then you can't profit the affiliations gave by US and you're requested to leave the area apace. The blueprints picked by can basically return henceforth once you have gotten a made demand from US through email. Call us : US : +1-888-254-4408 UK : +44-0808-234-2376 AUS : +61-1-800-985-062
    http://nortonsetup-norton-setup.com

    ReplyDelete
  59. Gmail is the best email service that has been presented by the web index mammoth Google with 15 GB of free storage room. It has made correspondence all the more simple and secure with its propelled highlights. Furthermore, this is the reason an ever increasing number of individuals are picking Gmail for their own and also business utilization step by step. Call us : US : +1-888-254-4408 UK : +44-0808-234-2376 AUS : +61-1-800-985-062
    http://gmail-gmail.com

    ReplyDelete

  60. A computer virus is a program that can copy itself and infect a computer without the permission or knowledge of the user.you can be sure of the privacy of your important files as well as confidential information.
    norton com setup

    ReplyDelete
  61. We are providing a step by step guide to set up Office along with independent support service for installation and activation issues.

    www.office.com/setup

    ReplyDelete