NVIDIA Nvenc Obs Guide

By Gerardo Delgado Cabrera on February 26, 2019 | Broadcasting Guides Streaming

About This Guide

The objective of this guide is to help you understand how to use the NVIDIA encoder, NVENC, in OBS. We have simplified some of the concepts to make this accessible to a wider audience. If you think we can improve any part of this guide or find any issues or mistakes, please post below and we will be happy to update it.

Bitrate, Resolution and Framerate

Encoding is all about compressing images. The smaller the size of the image, the less we must compress it and the more quality it keeps. While the same applies for framerate, a viewer can really notice a drop in FPS but not so much in resolution, so we will always try to stream at 60 FPS.

First, run a speed test to determine your upload speed (e.g. Speed Test). We want to use around 75% of your upload speed, as the game and other programs such as Discord will also fight for bandwidth.

Then, we will determine the resolution and FPS that we can use for such bitrate. Most streaming sites have recommendations (Twitch, Youtube, Facebook Gaming) on what to use. These are ours:

Upload Speed

Bitrate

Resolution

Framerate

3 Mbps

2,500

1024x576

30

4 Mbps

3,500

1280x720

30

6 Mbps

5,000

1280x720

60

8-10 Mbps

6,000

1920x1080 *

60

15+ Mbps

12,000 (Youtube)

1920x1080

60

20+ Mbps

15,000+ (Youtube)

2560x1440

60

40+ Mbps

30,000+ (Youtube)

3840x2160

60

* Important Note for High Motion Content. If you are going to stream high motion scenes (i.e. Racing games, some Battle Royale games, etc.) we highly recommend reducing your resolution. High motion content cannot be compressed as much, and can suffer from more artifacting (encoding errors) that make your stream look “blocky”. If you reduce the resolution, you reduce the data being encoded, and the resulting viewer quality is higher. For example, for Fortnite, many streamers decide to stream at 1600x900 60 FPS.

Note for New and Upcoming Streamers to Twitch. Transcoding allows a viewer to view your video on a different resolution, thus requiring a lower bandwidth. Twitch only offers guaranteed transcoding to Partners; non-partners may receive transcoding, but it is not guaranteed. This is important if your viewers are on mobile phones or their internet speed is not as fast. You may want to consider streaming at a lower bitrate and resolution to lower the bandwidth required to see your channel.

Recommended Settings

These are our recommended settings for OBS Studio 23.0 and up. You’ll want to test and adjust these settings by proceeding with a local recording to verify you’re happy with the results.

To access the settings, click on the Settings button on the bottom right.

Video Tab Settings

  • Base (Canvas) Resolution: Set the resolution you normally play at. That is, your desktop resolution (if you play in borderless mode), or the game resolution you normally enter (if you play in full screen).
  • Output (Scaled) Resolution: Enter the resolution appropriate for your Upload Speed and Bitrate, as we discussed in the previous section.
  • Downscale Filter: This allows you to select a downscale filter that will provide a small image sharpness enhancement, at the cost of some encoder workload. NVENC is very efficient and typically runs at low utilisation, so we recommend using this with the Lanczos, 36 samples option for the best quality.
  • FPS: Enter the FPS appropriate for your Upload Speed and Bitrate, as we discussed in the previous section.

Output Tab Settings

If you want an easy, out of the box configuration, then do the following:

  • Output Mode: Simple
  • Streaming:
    • Bitrate: Enter the Bitrate appropriate for your Upload Speed, as we discussed in the previous section.
    • Encoder: Select Hardware (NVENC).
    • Enable Advanced Encoder Settings: Unchecked. We collaborated with OBS to fine-tune these settings, so it just works!
    • Encoder Preset: Quality. This is already the default option. Note that it is only visible if you check Encode Advanced Encoder Settings.
  • Recording:
    • Recording Path: This is the directory where the videos will be saved. Make sure the hard drive you select has enough space!
    • Recording Quality: High Quality typically works for most users, but you can change this to Indistinguishable Quality if you have enough disk space or are going to do short videos (about 60 seconds).
    • Recording Format: FLV or MKV.
    • Encoder: Hardware (NVENC).

Other Settings

There are 2 other things you want to configure to ensure a smooth stream:

  • Windows: Make sure you update to Windows 10 version 1903, and enable Game Mode. This version includes performance enhancements for streaming, as well as an updated Game Mode compatible with streaming.
  • GPU Utilisation: If your GPU utilisation is above 95% Windows will start prioritising the game over everything; this can, in some cases, make your stream lag. To solve this, OBS added an option in OBS 24.0.3 to prioritise OBS Studio over the Game. Just run OBS as Admin, and your stream will be silky smooth.

If for some reason you don't want to run OBS in Admin mode, you can also limit your GPU usage to be below the 95% threshold. To do this, you can:

  • Cap FPS in-game, run the game in Borderless Windowed mode, reduce game graphics or resolution, or turn V-Sync on.
  • Run all assets at 1080p. To do that, double click on the source in OBS and under Resolution select Custom and specify a resolution equal or under 1080p.

And there you have it! We hope this helps you improve your stream quality and reach your goals. Leave us a comment if this worked for you or if you’d like us to update the guide with other info. Happy streaming!

Advanced Settings

If you want to mess around with all settings, here are our in-depth recommendations.

Streaming Settings

  • Output Mode: Advanced. This gives you access to all the settings. Let’s start!
  • Encoder: Select NVIDIA NVENC H.264 (new).
  • Enforce Streaming Service Encoder Settings: Leave this checked, this will ensure that if you enter a wrong value by mistake it gets corrected.
  • Rate Control: Select CBR. This determines the rate at which frames are going to be encoded.
  • Bitrate: Enter the bitrate appropriate for your Upload Speed, as we discussed in the previous section. Keep in mind that some platforms have a maximum bitrate (i.e. for Twitch it’s currently 6000 Kbps.).
  • Keyframe Interval: Set to 2. Streaming platforms may limit what you can select here, and most require a setting of 2.
  • Preset: Select Quality. You can change this to Max Quality to enable 2-pass encoding; this will provide you a minor quality increase but may cause problems in limited situations in maxed out GPUs.
  • Profile: Set to High. Profile determines a group of settings in the H.264 Codec. It doesn’t impact performance and gives access to a set of features that are key to streaming, so this should always be set to High.
  • Look-ahead: Checked. This allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify. B-frames are great because they increase image quality, but they consume a lot of your available bitrate, so they reduce quality on high motion content. Look-ahead enables the best of both worlds. This feature is CUDA accelerated; toggle this off if your GPU utilization is high to ensure a smooth stream.
  • Psycho Visual Tuning: Checked. This enables the Rate Distortion Optimization in the encoder, which greatly optimizes the way you use bitrate, improving image quality on movement.
  • GPU: 0. If you have 2 GPUs in your system, you can select which one is used to encode. This is not recommended, as NVENC is already very efficient and the little gain you can get from using a second card is lost by having to copy the frame to the second GPU.
  • Max B-Frames: Set to 4. If you uncheck the Look-ahead option, reduce this to 2 B-Frames.

Recording Settings

If you want to stream with HEVC, you will also have to adjust your Stream settings. Note that HEVC live streaming is currently only supported by YouTube. To stream to YouTube with HEVC, navigate to the Stream tab and change the Service to YouTube - HLS. You may need to click Show All… to find this option.

  • Type: Standard.
  • Recording Path: This is the directory where the videos will be saved. Make sure the hard drive you select has enough space!
  • Recording Format: FLV; or MKV if you use multiple audio tracks.
  • Audio Track: Leave it at 1 for default; you can add more audio tracks if you are using more sources.
  • Encoder: NVIDIA NVENC H.264 (new).
  • Rate Control: We recommend CQP, although VBR will also produce good results.
    • CQ Level (CQ): 15 (you can decrease the number to get higher quality).
    • Bitrate and Max Bitrate (VBR): 40,000 Bitrate; 60,000 Max bitrate. You can increase these to 100,000 and 200,000 (respectively) for higher quality.
  • Keyframe Interval: 0 or 2.
  • Preset: Select Quality. You can change this to Max-Quality to enable 2-pass encoding; this will provide you a minor quality increase but may cause problems in limited situations in maxed out GPUs.
  • Profile: Set to High.
  • Look-ahead: Checked.
  • Psycho Visual Tuning: Checked.
  • GPU: 0. If you have 2 GPUs in your system, you can select which one is used to encode.
  • Max B-Frames: Set to 4. If you uncheck the Look-ahead option, reduce this to 2 B-Frames.

Recording and Streaming HDR with Hevc and HDR

With OBS version 28, you can stream with HEVC to platforms that support it. HEVC is an improved codec that provides 15% better encoding efficiency than H.264 (i.e. it provides quality as if you streamed with 15% more bitrate). HEVC - and OBS version 28 - also allow you to record and stream HDR content.

HOW TO RECORD AND/OR STREAM WITH HEVC

If you want to record or stream with HEVC, in OBS navigate to your Output window and change the Encoder to Hardware (NVENC, HEVC) for the Streaming or Recording settings, as needed.

Screenshot1

If you want to stream with HEVC, you will also have to adjust your Stream settings. Note that HEVC live streaming is currently only supported by YouTube. To stream to YouTube with HEVC, navigate to the Stream tab and change the Service to YouTube - HLS. You may need to click Show All… to find this option.

Screenshot2

Now you should be able to connect your YouTube account.

Finally, navigate to your YouTube Live settings on YouTube.com, and make sure that your Latency setting is set to Low or Normal (do not use Ultra Low as it is not compatible).

Screenshot3

Streaming or Recording in HDR

HDR - or High Dynamic Range - allows you to increase the range of light of your content. You can record and/or stream with HDR. 

Note that:

  • HDR content can look washed off in non-HDR displays, so you don’t want to turn this on if your display does not support HDR, or if the platform you are streaming to does not support HDR.
  • HDR requires HEVC; it’s not compatible with H.264.

In order to enable HDR, we’ll need to make sure that:

  1. You have an HDR monitor.
  2. Your Windows and Game settings have HDR turned on.
  3. We configure your full pipeline in OBS to enable HDR.
Windows and Game Settings
 
Start by making sure that you have HDR turned on for your Display in Windows. Right Click on your desktop > Display Settings, and turn on Use HDR.
Screenshot4

Next, open the game you want to record or stream, go to the settings and make sure HDR is turned on. In Windows 11 there’s a feature called Auto HDR that enables a virtual HDR mode for any game. This is also a valid option.

OBS Settings

In OBS, start by going to the Advanced tab and changing the Color Format to P010 and the Color Space to Rec. 2100 (PQ).

Screenshot5

Next, we need to adjust the Game Capture source. Double click it (or right click it and go to Properties), scroll down find the RGB10A2 Color Space. Make sure this is set to Rec. 2100 (PQ).

Screenshot6

Your content should now be ready to be captured and/or streamed in HDR! But remember, in order to record and/or stream in HDR you need to change your encoder from H.264 to HEVC. You can visit the How to Record and/or Stream with HEVC section above to adjust this.

About NVENC

NVENC is NVIDIA’s encoder. It’s a physical section of our GPUs that is dedicated to encoding only. This means that your GPU can operate normally regardless of whether you use this region to stream or record. Other encoders, such as x264, use your CPU to encode, which takes resources away from other programs such as your game. That’s why using NVENC allows you to play games at a higher framerate and avoid stuttering, giving you and your viewers a better experience.

In the last two GPU generations we have made great improvements to NVENC, helping deliver best-in-class output quality. NVENC in the GTX 10-series GPUs provides superior quality than x264 Very Fast, the most commonly used x264 preset. And in the new RTX 20 and 30-series, NVENC performs better than x264 Fast and on par with x264 Medium, a preset that requires an expensive dual PC setup.

One thing that is great about NVENC on the GeForce RTX 20 and 30-series and GeForce GTX 1650 Super and up is that all GPUs have the same NVENC with the same performance and quality, from the RTX 2060 to the RTX 3090. NVENC also benefits from our own NVIDIA Video Codec SDK, an advanced set of tools that help improve the encoded quality and that we constantly update to help you get the best out of your NVIDIA card.

Finally, if you are using an NVIDIA GPU you have access to GeForce Experience’s Game Filters, which allow you to further improve the image quality of your viewers via software by enhancing color, adding sharpness, or introducing cool effects.

About NVIDIA Broadcast

NVIDIA Broadcast transforms your gamer den into a home broadcast studio, upgrading standard webcams and microphones into premium, smart devices through the power of AI. Improve the video and audio quality of your livestream through AI capabilities such as virtual background or webcam auto frame, and microphone noise removal. With dedicated AI processors called Tensor Cores on GeForce RTX GPUs, the AI networks are able to run high-quality effects in real-time.

NVIDIA Broadcast features include:

  • Noise Removal: use AI to remove background noise from your microphone feed – be it a loud mechanical keyboard or the AC. The AI network can even be used on incoming audio feeds to mute that one friend who won’t turn on push-to-talk.
  • Virtual Background: use AI to remove the background of your webcam feed and replace it with game footage, a replacement image, or even a subtle blur.
  • Auto Frame: use AI to track your head movement, zooming and automatically cropping on your head so you are always in the frame, even as you move around.

When not broadcasting, these features can also enhance your video conference calls and voice chats, making NVIDIA Broadcast a perfect AI companion for broadcasters and gamers alike.

NVIDIA Broadcast is compatible with a wide range of broadcasting, video conferencing and voice chat apps. These include OBS Studio, Streamlabs, Xsplit, Discord, Skype, Zoom, Webex, Microsoft Teams, Google Meet and more.

About the GeForce-Optimised New Version of OBS Studio

We have collaborated with OBS to improve support for NVIDIA GeForce GPUs. The new OBS Studio, version 23.0, will leverage the NVIDIA Video Codec SDK, which will greatly improve performance and reduce the FPS impact of streaming and recording. We have also tweaked some of the background settings of NVENC to improve quality, especially for the RTX 20-Series GPUs.

How to Debug Problems

Streaming can be very complicated, but it’s particularly hard to debug. There are many things at play when you stream, so we are going to try to provide you some help on how to identify what is going wrong and how to fix it.

Components

Streaming uses the following components:

  • Your PC: This includes hardware and software.
  • Local Internet: WiFi or cabled internet + your Router.
  • Your connection: To your service provider.
  • The platform: Twitch, Youtube, Facebook Gaming, etc.
  • Viewer’s Internet: Typically Wi-Fi, but can also be 3G/4G.
  • Viewer’s device: keep in mind 35% of Twitch viewers are on mobile.

If something is failing, we want to first identify what component may be failing, so we don’t go crazy trying to fix something that was never broken in the first place. Typically, this means that the first test you should do is a Speed Test to make sure that you don’t have internet problems in your local internet or your connection. Second, make sure the platform hasn’t issued an alert that they are down or are experiencing problems. Then based on what error you get, you start looking at one thing or another in your PC.

How to check what’s happening to the encode

OBS Studio includes a very useful tool: the Stats Window. To bring it up, click on View > Stats. This window will show you Lagged and Skipped frames, Dropped frames, and Encode FPS.

This window will show you:

  • FPS at which you are encoding.
  • Latency to encode each frame.
  • Missed Frames - problems with GPU.
  • Skipped Frames - problems with CPU.
  • Dropped Frames - problems with network.

Common Error Types

Stream is missing FPS. The stats window will show missed frames. While streaming and missing frames, pull up the Task Manager > Performance, Click on GPU and check the 3D load and Encoder load.

Solutions:

  • If the 3D load is above 95%, especially at 1440p or 4K setups, Windows may be prioritising the Game over OBS. To fix this, we have a special mode inserted on OBS 24.0.3 where you can prioritize OBS above the Game. Just run OBS in Game Mode.
  • If the Video Encode load is maxed out, we need to lower the load. NVENC can do up to 8K30, so the only way to overload it is to do 2x4K60 streams. If you are encoding 4K60, make sure that your quality setting in OBS is set to Quality, not Max Quality. Max Quality does 2 pass encoding (i.e. encodes twice), which is too much for the encoder.

Image looks very washed out. The most likely issue is trying to push too much quality with not enough bitrate. Consider reducing the resolution, and frame rate (if needed), and try again. If quality improves, then adjust until you find your sweet spot.