Log into the web portal for office.
Click on your initials in the top corner

Select View account
Under Security Info select update info
Select Add Method
From the drop down pick Authenticator app
Select I want to use a different authenticator app
You will be prompted to setup a new account in your app. In the yubico authenticator click the vertical elipsis point and then select add account
Please note that at the time of writing the option Scan QR code does not work.
On the office website you will now select "Can't scan image"
This will add several lines to the page as seen here:
The account name and Secret key both have a quick copy button to the right side of them.
Back on the authenticator application we add the information like so
Issuer can be named anything but for a standard we've been using MS Office.
After the account has been added to the Yubico authenticator click next on the website. You will be prompted for a six digit code
Back on the Yubico authenticator you will be given a six digit code that resets every 30 seconds
Note the clock on the right hand side that indicates how long the code has before a new one will be given.
Double clicking the code will copy it to your clip board
Paste or write it into the field and click next to complete the set up.
With that this authentication method is now activated.